[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:31:58 dist-test-slave-qcpv krb5kdc[29508](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:31:58 dist-test-slave-qcpv krb5kdc[29508](info): set up 2 sockets
Sep 29 18:31:58 dist-test-slave-qcpv krb5kdc[29508](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 18:31:59.404237 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.001s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:31:59 dist-test-slave-qcpv krb5kdc[29508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:31:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:31:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.207.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
I20260929 18:31:59.561322 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:43355
--webserver_interface=127.28.207.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:33979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:43355
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:31:59.680944 29521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:31:59.681310 29521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:31:59.681401 29521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:31:59.685678 29521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:31:59.685802 29521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:31:59.685838 29521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:31:59.685875 29521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:31:59.685910 29521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:31:59.692473 29521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:33979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:43355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:43355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:31:59.693742 29521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:31:59.694837 29521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:31:59.706072 29527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:59.706197 29521 server_base.cc:1034] running on GCE node
W20260929 18:31:59.706072 29529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:31:59.706411 29526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:31:59.707356 29521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:31:59.708963 29521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:31:59.710232 29521 hybrid_clock.cc:648] HybridClock initialized: now 1790706719710192 us; error 61 us; skew 500 ppm
Sep 29 18:31:59 dist-test-slave-qcpv krb5kdc[29508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706719, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:31:59.714352 29521 init.cc:377] Logged in from keytab as kudu/127.28.207.126@KRBTEST.COM (short username kudu)
I20260929 18:31:59.715689 29521 webserver.cc:460] Webserver started at http://127.28.207.126:46797/ using document root <none> and password file <none>
I20260929 18:31:59.716463 29521 fs_manager.cc:359] Metadata directory not provided
I20260929 18:31:59.716579 29521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:31:59.716874 29521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:31:59.719602 29521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "574421a1fda842ef947c67e8bf134da4"
format_stamp: "Formatted at 2026-09-29 18:31:59 on dist-test-slave-qcpv"
server_key: "b4cbf7f63bc90f7c8e38241477e83f14"
server_key_iv: "6aff5f4d9e13947cbf91d5c07aa169da"
server_key_version: "encryptionkey@0"
I20260929 18:31:59.720361 29521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "574421a1fda842ef947c67e8bf134da4"
format_stamp: "Formatted at 2026-09-29 18:31:59 on dist-test-slave-qcpv"
server_key: "b4cbf7f63bc90f7c8e38241477e83f14"
server_key_iv: "6aff5f4d9e13947cbf91d5c07aa169da"
server_key_version: "encryptionkey@0"
I20260929 18:31:59.725510 29521 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 18:31:59.729354 29536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:31:59.731307 29521 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:31:59.731531 29521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "574421a1fda842ef947c67e8bf134da4"
format_stamp: "Formatted at 2026-09-29 18:31:59 on dist-test-slave-qcpv"
server_key: "b4cbf7f63bc90f7c8e38241477e83f14"
server_key_iv: "6aff5f4d9e13947cbf91d5c07aa169da"
server_key_version: "encryptionkey@0"
I20260929 18:31:59.731711 29521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:31:59.771430 29521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:31:59.776257 29521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:31:59.776567 29521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:31:59.787420 29521 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:43355
I20260929 18:31:59.787563 29588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:43355 every 8 connection(s)
I20260929 18:31:59.788897 29521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:31:59.791826 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 29521
I20260929 18:31:59.792070 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:31:59.792639 29589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:31:59.793797 29501 external_mini_cluster.cc:1428] Setting key 9ee1dddc11e32556a4120e3e5dc2153e
I20260929 18:31:59.799235 29589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 574421a1fda842ef947c67e8bf134da4: Bootstrap starting.
I20260929 18:31:59.801923 29589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 574421a1fda842ef947c67e8bf134da4: Neither blocks nor log segments found. Creating new log.
I20260929 18:31:59.803043 29589 log.cc:824] T 00000000000000000000000000000000 P 574421a1fda842ef947c67e8bf134da4: Log is configured to *not* fsync() on all Append() calls
I20260929 18:31:59.806042 29589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 574421a1fda842ef947c67e8bf134da4: No bootstrap required, opened a new log
I20260929 18:31:59.810796 29589 raft_consensus.cc:348] T 00000000000000000000000000000000 P 574421a1fda842ef947c67e8bf134da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "574421a1fda842ef947c67e8bf134da4" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43355 } }
I20260929 18:31:59.811179 29589 raft_consensus.cc:374] T 00000000000000000000000000000000 P 574421a1fda842ef947c67e8bf134da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:31:59.811246 29589 raft_consensus.cc:729] T 00000000000000000000000000000000 P 574421a1fda842ef947c67e8bf134da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 574421a1fda842ef947c67e8bf134da4, State: Initialized, Role: FOLLOWER
I20260929 18:31:59.811812 29589 consensus_queue.cc:260] T 00000000000000000000000000000000 P 574421a1fda842ef947c67e8bf134da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "574421a1fda842ef947c67e8bf134da4" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43355 } }
I20260929 18:31:59.811990 29589 raft_consensus.cc:388] T 00000000000000000000000000000000 P 574421a1fda842ef947c67e8bf134da4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:31:59.812091 29589 raft_consensus.cc:482] T 00000000000000000000000000000000 P 574421a1fda842ef947c67e8bf134da4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:31:59.812223 29589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 574421a1fda842ef947c67e8bf134da4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:31:59.813649 29589 raft_consensus.cc:504] T 00000000000000000000000000000000 P 574421a1fda842ef947c67e8bf134da4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "574421a1fda842ef947c67e8bf134da4" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43355 } }
I20260929 18:31:59.814065 29589 leader_election.cc:302] T 00000000000000000000000000000000 P 574421a1fda842ef947c67e8bf134da4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 574421a1fda842ef947c67e8bf134da4; no voters:
I20260929 18:31:59.814481 29589 leader_election.cc:288] T 00000000000000000000000000000000 P 574421a1fda842ef947c67e8bf134da4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:31:59.814767 29594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 574421a1fda842ef947c67e8bf134da4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:31:59.815193 29594 raft_consensus.cc:686] T 00000000000000000000000000000000 P 574421a1fda842ef947c67e8bf134da4 [term 1 LEADER]: Becoming Leader. State: Replica: 574421a1fda842ef947c67e8bf134da4, State: Running, Role: LEADER
I20260929 18:31:59.815748 29594 consensus_queue.cc:237] T 00000000000000000000000000000000 P 574421a1fda842ef947c67e8bf134da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "574421a1fda842ef947c67e8bf134da4" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43355 } }
I20260929 18:31:59.816025 29589 sys_catalog.cc:564] T 00000000000000000000000000000000 P 574421a1fda842ef947c67e8bf134da4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:31:59.818359 29595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 574421a1fda842ef947c67e8bf134da4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "574421a1fda842ef947c67e8bf134da4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "574421a1fda842ef947c67e8bf134da4" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43355 } } }
I20260929 18:31:59.818567 29595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 574421a1fda842ef947c67e8bf134da4 [sys.catalog]: This master's current role is: LEADER
I20260929 18:31:59.818965 29596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 574421a1fda842ef947c67e8bf134da4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 574421a1fda842ef947c67e8bf134da4. Latest consensus state: current_term: 1 leader_uuid: "574421a1fda842ef947c67e8bf134da4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "574421a1fda842ef947c67e8bf134da4" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43355 } } }
I20260929 18:31:59.819068 29596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 574421a1fda842ef947c67e8bf134da4 [sys.catalog]: This master's current role is: LEADER
Sep 29 18:31:59 dist-test-slave-qcpv krb5kdc[29508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
W20260929 18:31:59.822995 29609 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 574421a1fda842ef947c67e8bf134da4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:31:59.823104 29609 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:31:59.823210 29608 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:31:59.823936 29608 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:31:59.829380 29591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:31:59.798036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42648 (local address 127.28.207.126:43355)
0929 18:31:59.798739 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0929 18:31:59.798760 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 18:31:59.800007 (+ 1247us) server_negotiation.cc:407] Connection header received
0929 18:31:59.804396 (+ 4389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:31:59.804425 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:31:59.804912 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:31:59.805371 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:31:59.809272 (+ 3901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:59.811077 (+ 1805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:59.812967 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:31:59.813938 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:31:59.814165 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:31:59.821501 (+ 7336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:31:59.821511 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:31:59.821537 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:31:59.821589 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:31:59.824146 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:59.824922 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:59.824929 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:59.824942 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:59.825095 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:31:59.825541 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:31:59.825545 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:31:59.825547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:31:59.826104 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:31:59.826344 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 18:31:59.827373 (+ 1029us) server_negotiation.cc:299] Negotiation successful
0929 18:31:59.827911 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":217,"threads_started":1}
I20260929 18:31:59.830749 29608 catalog_manager.cc:1347] Generated new cluster ID: a691ffbfb8384a0283a57e54ab0dbd1f
I20260929 18:31:59.830832 29608 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:31:59.851534 29608 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:31:59.853166 29608 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:31:59.873623 29608 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 574421a1fda842ef947c67e8bf134da4: Generated new TSK 0
I20260929 18:31:59.874702 29608 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.207.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab.
Entry for principal kudu/127.28.207.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab.
I20260929 18:31:59.920783 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.65:0
--local_ip_for_outbound_sockets=127.28.207.65
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:43355
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:33979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:00.055510 29615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:00.055841 29615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:00.055891 29615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:00.060804 29615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:00.060916 29615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:00.060985 29615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.65
I20260929 18:32:00.067001 29615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:33979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:43355
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.65
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:00.068188 29615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:00.069528 29615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:00.081139 29621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:00.081283 29620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:00.081427 29615 server_base.cc:1034] running on GCE node
W20260929 18:32:00.081261 29623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:00.082115 29615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:00.082919 29615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:00.086268 29615 hybrid_clock.cc:648] HybridClock initialized: now 1790706720086229 us; error 64 us; skew 500 ppm
Sep 29 18:32:00 dist-test-slave-qcpv krb5kdc[29508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706720, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:00.090027 29615 init.cc:377] Logged in from keytab as kudu/127.28.207.65@KRBTEST.COM (short username kudu)
I20260929 18:32:00.091269 29615 webserver.cc:460] Webserver started at http://127.28.207.65:37671/ using document root <none> and password file <none>
I20260929 18:32:00.091915 29615 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:00.092016 29615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:00.092242 29615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:00.094673 29615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-0/data/instance:
uuid: "3ca68af093c3419eb4322dd7ae8eed6a"
format_stamp: "Formatted at 2026-09-29 18:32:00 on dist-test-slave-qcpv"
server_key: "3d06baddf618f50c62ff63671791b8a6"
server_key_iv: "ee11436413932632ebbe11de4631c535"
server_key_version: "encryptionkey@0"
I20260929 18:32:00.095428 29615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance:
uuid: "3ca68af093c3419eb4322dd7ae8eed6a"
format_stamp: "Formatted at 2026-09-29 18:32:00 on dist-test-slave-qcpv"
server_key: "3d06baddf618f50c62ff63671791b8a6"
server_key_iv: "ee11436413932632ebbe11de4631c535"
server_key_version: "encryptionkey@0"
I20260929 18:32:00.100212 29615 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:32:00.103308 29630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:00.105178 29615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 18:32:00.105373 29615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-0/wal
uuid: "3ca68af093c3419eb4322dd7ae8eed6a"
format_stamp: "Formatted at 2026-09-29 18:32:00 on dist-test-slave-qcpv"
server_key: "3d06baddf618f50c62ff63671791b8a6"
server_key_iv: "ee11436413932632ebbe11de4631c535"
server_key_version: "encryptionkey@0"
I20260929 18:32:00.105518 29615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:00.118844 29615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:00.122746 29615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:00.123028 29615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:00.123693 29615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:32:00.124609 29615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:00.124689 29615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:00.124758 29615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:00.124812 29615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:00.136834 29615 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.65:35453
I20260929 18:32:00.137260 29743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.65:35453 every 8 connection(s)
I20260929 18:32:00.139467 29615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
I20260929 18:32:00.140384 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 29615
I20260929 18:32:00.140527 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance
I20260929 18:32:00.140904 29501 external_mini_cluster.cc:1428] Setting key 172c90f7dc32df2648d5494d3dbb928c
WARNING: no policy specified for kudu/127.28.207.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.66@KRBTEST.COM" created.
Sep 29 18:32:00 dist-test-slave-qcpv krb5kdc[29508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706720, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:32:00.167088 29591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:00.142803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:50399 (local address 127.28.207.126:43355)
0929 18:32:00.143133 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:00.143138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:00.146954 (+ 3816us) server_negotiation.cc:407] Connection header received
0929 18:32:00.148633 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:00.148637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:00.148711 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:00.148830 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:00.150722 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.151549 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.153701 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.153988 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.154096 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:00.161520 (+ 7424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:00.161530 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:00.161536 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:00.161600 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:00.163625 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:00.164572 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:00.164576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:00.164577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:00.164803 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:00.165371 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:00.165375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:00.165376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:00.165646 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:00.165751 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:00.166630 (+ 879us) server_negotiation.cc:299] Negotiation successful
0929 18:32:00.166812 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220}
I20260929 18:32:00.168296 29746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:00.145061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43355 (local address 127.28.207.65:50399)
0929 18:32:00.146679 (+ 1618us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:00.146787 (+ 108us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:00.148273 (+ 1486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:00.149095 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:00.149110 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:00.149739 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:00.150483 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:00.150506 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.151725 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.151741 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:00.153475 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:00.153491 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.154202 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.154209 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:00.154443 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:00.155310 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:00.155338 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:00.161150 (+ 5812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:00.163841 (+ 2691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:00.163853 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:00.163877 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:00.164368 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:00.164964 (+ 596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:00.164970 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:00.164973 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:00.165204 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:00.165812 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:00.165823 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:00.166261 (+ 438us) client_negotiation.cc:769] Sending connection context
0929 18:32:00.166541 (+ 280us) client_negotiation.cc:240] Negotiation successful
0929 18:32:00.167030 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1216,"thread_start_us":926,"threads_started":1}
I20260929 18:32:00.170442 29744 heartbeater.cc:347] Connected to a master server at 127.28.207.126:43355
I20260929 18:32:00.170842 29744 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:00.171579 29744 heartbeater.cc:511] Master 127.28.207.126:43355 requested a full tablet report, sending...
I20260929 18:32:00.173676 29553 ts_manager.cc:194] Registered new tserver with Master: 3ca68af093c3419eb4322dd7ae8eed6a (127.28.207.65:35453)
I20260929 18:32:00.175170 29553 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.65@KRBTEST.COM'} at 127.28.207.65:50399
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab.
Entry for principal kudu/127.28.207.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab.
I20260929 18:32:00.180397 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.66:0
--local_ip_for_outbound_sockets=127.28.207.66
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:43355
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:33979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:00.294706 29749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:00.295045 29749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:00.295095 29749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:00.299281 29749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:00.299356 29749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:00.299420 29749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.66
I20260929 18:32:00.304808 29749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:33979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:43355
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.66
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:00.305914 29749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:00.307072 29749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:00.319140 29754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:00.319070 29755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:00.319290 29757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:00.320775 29749 server_base.cc:1034] running on GCE node
I20260929 18:32:00.321231 29749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:00.321980 29749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:00.323195 29749 hybrid_clock.cc:648] HybridClock initialized: now 1790706720323155 us; error 61 us; skew 500 ppm
Sep 29 18:32:00 dist-test-slave-qcpv krb5kdc[29508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706720, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:00.326493 29749 init.cc:377] Logged in from keytab as kudu/127.28.207.66@KRBTEST.COM (short username kudu)
I20260929 18:32:00.327584 29749 webserver.cc:460] Webserver started at http://127.28.207.66:45653/ using document root <none> and password file <none>
I20260929 18:32:00.328215 29749 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:00.328315 29749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:00.328548 29749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:00.330682 29749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-1/data/instance:
uuid: "86fe7e62fdf34cf29a469044a4ac8a69"
format_stamp: "Formatted at 2026-09-29 18:32:00 on dist-test-slave-qcpv"
server_key: "76be3719238effb208af899430102afc"
server_key_iv: "9946bf138d0415dd33f2e5de5cfc0fde"
server_key_version: "encryptionkey@0"
I20260929 18:32:00.331329 29749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance:
uuid: "86fe7e62fdf34cf29a469044a4ac8a69"
format_stamp: "Formatted at 2026-09-29 18:32:00 on dist-test-slave-qcpv"
server_key: "76be3719238effb208af899430102afc"
server_key_iv: "9946bf138d0415dd33f2e5de5cfc0fde"
server_key_version: "encryptionkey@0"
I20260929 18:32:00.336301 29749 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 18:32:00.339449 29764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:00.341070 29749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:32:00.341236 29749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-1/wal
uuid: "86fe7e62fdf34cf29a469044a4ac8a69"
format_stamp: "Formatted at 2026-09-29 18:32:00 on dist-test-slave-qcpv"
server_key: "76be3719238effb208af899430102afc"
server_key_iv: "9946bf138d0415dd33f2e5de5cfc0fde"
server_key_version: "encryptionkey@0"
I20260929 18:32:00.341374 29749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:00.356583 29749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:00.360039 29749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:00.360255 29749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:00.360870 29749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:32:00.361711 29749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:00.361785 29749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:00.361858 29749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:00.361904 29749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:00.372618 29749 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.66:34723
I20260929 18:32:00.372880 29877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.66:34723 every 8 connection(s)
I20260929 18:32:00.373672 29749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
I20260929 18:32:00.379756 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 29749
I20260929 18:32:00.379923 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance
I20260929 18:32:00.380229 29501 external_mini_cluster.cc:1428] Setting key 5c941d3309a4d5982285a3be1a3a00d6
Sep 29 18:32:00 dist-test-slave-qcpv krb5kdc[29508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706720, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.207.67@KRBTEST.COM; defaulting to no policy
I20260929 18:32:00.394270 29591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:00.376498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:44495 (local address 127.28.207.126:43355)
0929 18:32:00.376664 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:00.376668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:00.377743 (+ 1075us) server_negotiation.cc:407] Connection header received
0929 18:32:00.379657 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:00.379662 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:00.379755 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:00.379899 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:00.381709 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.382435 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.384752 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.385176 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.385327 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:00.389177 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:00.389182 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:00.389185 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:00.389223 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:00.391345 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:00.392233 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:00.392241 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:00.392244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:00.392358 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:00.392824 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:00.392827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:00.392829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:00.393064 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:00.393153 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:00.393925 (+ 772us) server_negotiation.cc:299] Negotiation successful
0929 18:32:00.394088 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.67@KRBTEST.COM" created.
I20260929 18:32:00.395313 29880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:00.376808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43355 (local address 127.28.207.66:44495)
0929 18:32:00.377431 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:00.377494 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:00.379294 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:00.380158 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:00.380172 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:00.380773 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:00.381523 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:00.381545 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.382605 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.382610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:00.384529 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:00.384547 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.385436 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.385441 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:00.385670 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:00.386465 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:00.386487 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:00.388843 (+ 2356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:00.391533 (+ 2690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:00.391545 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:00.391567 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:00.391982 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:00.392501 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:00.392503 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:00.392505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:00.392650 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:00.393199 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:00.393204 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:00.393609 (+ 405us) client_negotiation.cc:769] Sending connection context
0929 18:32:00.393792 (+ 183us) client_negotiation.cc:240] Negotiation successful
0929 18:32:00.394126 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":137,"threads_started":1}
I20260929 18:32:00.396958 29878 heartbeater.cc:347] Connected to a master server at 127.28.207.126:43355
I20260929 18:32:00.397539 29878 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:00.398625 29878 heartbeater.cc:511] Master 127.28.207.126:43355 requested a full tablet report, sending...
I20260929 18:32:00.400422 29553 ts_manager.cc:194] Registered new tserver with Master: 86fe7e62fdf34cf29a469044a4ac8a69 (127.28.207.66:34723)
I20260929 18:32:00.401394 29553 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.66@KRBTEST.COM'} at 127.28.207.66:44495
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab.
Entry for principal kudu/127.28.207.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab.
I20260929 18:32:00.415880 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.67:0
--local_ip_for_outbound_sockets=127.28.207.67
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:43355
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:33979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:00.529134 29883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:00.529439 29883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:00.529510 29883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:00.534291 29883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:00.534384 29883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:00.534449 29883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.67
I20260929 18:32:00.540536 29883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:33979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:43355
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.67
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:00.541646 29883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:00.542644 29883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:00.553853 29888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:00.554039 29889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:00.554042 29891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:00.554111 29883 server_base.cc:1034] running on GCE node
I20260929 18:32:00.554755 29883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:00.555586 29883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:00.556947 29883 hybrid_clock.cc:648] HybridClock initialized: now 1790706720556902 us; error 61 us; skew 500 ppm
Sep 29 18:32:00 dist-test-slave-qcpv krb5kdc[29508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706720, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:00.560724 29883 init.cc:377] Logged in from keytab as kudu/127.28.207.67@KRBTEST.COM (short username kudu)
I20260929 18:32:00.561805 29883 webserver.cc:460] Webserver started at http://127.28.207.67:41209/ using document root <none> and password file <none>
I20260929 18:32:00.562464 29883 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:00.562577 29883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:00.562817 29883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:00.564955 29883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-2/data/instance:
uuid: "14fbda73e512462f988640de5bdadcf3"
format_stamp: "Formatted at 2026-09-29 18:32:00 on dist-test-slave-qcpv"
server_key: "8497ae4864862705526ae79db824683f"
server_key_iv: "1da9c084843da6c7a350d229f04c4b72"
server_key_version: "encryptionkey@0"
I20260929 18:32:00.565547 29883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance:
uuid: "14fbda73e512462f988640de5bdadcf3"
format_stamp: "Formatted at 2026-09-29 18:32:00 on dist-test-slave-qcpv"
server_key: "8497ae4864862705526ae79db824683f"
server_key_iv: "1da9c084843da6c7a350d229f04c4b72"
server_key_version: "encryptionkey@0"
I20260929 18:32:00.569736 29883 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260929 18:32:00.572904 29898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:00.574585 29883 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 18:32:00.574770 29883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-2/wal
uuid: "14fbda73e512462f988640de5bdadcf3"
format_stamp: "Formatted at 2026-09-29 18:32:00 on dist-test-slave-qcpv"
server_key: "8497ae4864862705526ae79db824683f"
server_key_iv: "1da9c084843da6c7a350d229f04c4b72"
server_key_version: "encryptionkey@0"
I20260929 18:32:00.574934 29883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:00.598675 29883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:00.602465 29883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:00.602715 29883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:00.603480 29883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:32:00.604394 29883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:00.604476 29883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:00.604550 29883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:00.604606 29883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:00.615403 29883 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.67:38451
I20260929 18:32:00.615520 30011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.67:38451 every 8 connection(s)
I20260929 18:32:00.616519 29883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
I20260929 18:32:00.624507 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 29883
I20260929 18:32:00.624706 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance
I20260929 18:32:00.625077 29501 external_mini_cluster.cc:1428] Setting key aebd84624eac0d2f7840cdb7920e4215
Sep 29 18:32:00 dist-test-slave-qcpv krb5kdc[29508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706720, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:32:00.635215 29591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:00.619769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:37737 (local address 127.28.207.126:43355)
0929 18:32:00.620018 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:00.620022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:00.621111 (+ 1089us) server_negotiation.cc:407] Connection header received
0929 18:32:00.622570 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:00.622573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:00.622636 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:00.622722 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:00.624015 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.624639 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.626633 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.626989 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.627100 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:00.630718 (+ 3618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:00.630721 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:00.630724 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:00.630752 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:00.632517 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:00.633192 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:00.633195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:00.633196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:00.633278 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:00.633813 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:00.633816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:00.633818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:00.634048 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:00.634146 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:00.634967 (+ 821us) server_negotiation.cc:299] Negotiation successful
0929 18:32:00.635080 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260929 18:32:00.636689 30014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:00.620245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43355 (local address 127.28.207.67:37737)
0929 18:32:00.620897 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:00.620951 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:00.622313 (+ 1362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:00.622967 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:00.622975 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:00.623366 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:00.623859 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:00.623872 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.624849 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.624855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:00.626433 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:00.626450 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.627197 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.627202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:00.627450 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:00.628403 (+ 953us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:00.628422 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:00.630469 (+ 2047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:00.632688 (+ 2219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:00.632694 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:00.632714 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:00.633065 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:00.633430 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:00.633433 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:00.633435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:00.633642 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:00.634194 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:00.634202 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:00.634563 (+ 361us) client_negotiation.cc:769] Sending connection context
0929 18:32:00.634792 (+ 229us) client_negotiation.cc:240] Negotiation successful
0929 18:32:00.635425 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":169,"threads_started":1}
I20260929 18:32:00.638790 30012 heartbeater.cc:347] Connected to a master server at 127.28.207.126:43355
I20260929 18:32:00.639271 30012 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:00.640388 30012 heartbeater.cc:511] Master 127.28.207.126:43355 requested a full tablet report, sending...
I20260929 18:32:00.641938 29552 ts_manager.cc:194] Registered new tserver with Master: 14fbda73e512462f988640de5bdadcf3 (127.28.207.67:38451)
I20260929 18:32:00.642669 29552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.67@KRBTEST.COM'} at 127.28.207.67:37737
I20260929 18:32:00.655339 29501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 18:32:00 dist-test-slave-qcpv krb5kdc[29508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706720, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 18:32:00 dist-test-slave-qcpv krb5kdc[29508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706720, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.207.65@KRBTEST.COM
I20260929 18:32:00.692610 30019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:00.679564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45490 (local address 127.28.207.65:35453)
0929 18:32:00.679955 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:00.679962 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:00.679987 (+ 25us) server_negotiation.cc:407] Connection header received
0929 18:32:00.680062 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:00.680067 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:00.680265 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:00.680398 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:00.681464 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.682269 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.683437 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.683861 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.683994 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:00.687750 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:00.687762 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:00.687778 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:00.687812 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:00.689937 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:00.690631 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:00.690637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:00.690638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:00.690743 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:00.691330 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:00.691333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:00.691335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:00.691630 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:00.691813 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:00.692230 (+ 417us) server_negotiation.cc:299] Negotiation successful
0929 18:32:00.692384 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":135,"threads_started":1}
W20260929 18:32:00.694308 29655 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45490
Sep 29 18:32:00 dist-test-slave-qcpv krb5kdc[29508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 18:32:00 dist-test-slave-qcpv krb5kdc[29508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.207.65@KRBTEST.COM
I20260929 18:32:00.729784 30019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:00.717191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45494 (local address 127.28.207.65:35453)
0929 18:32:00.717342 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:00.717348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:00.717604 (+ 256us) server_negotiation.cc:407] Connection header received
0929 18:32:00.717791 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:00.717794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:00.717845 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:00.717940 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:00.719074 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.719617 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.720727 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:00.720953 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:00.721050 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:00.725977 (+ 4927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:00.725980 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:00.725983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:00.726011 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:00.727684 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:00.728277 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:00.728280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:00.728282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:00.728358 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:00.728732 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:00.728734 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:00.728735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:00.728900 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:00.729040 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:00.729405 (+ 365us) server_negotiation.cc:299] Negotiation successful
0929 18:32:00.729582 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 18:32:00.731594 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 29615
I20260929 18:32:00.739418 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 29749
I20260929 18:32:00.746485 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 29883
I20260929 18:32:00.752980 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 29521
2026-09-29T18:32:00Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2487 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:32:00 dist-test-slave-qcpv krb5kdc[30026](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:32:00 dist-test-slave-qcpv krb5kdc[30026](info): set up 2 sockets
Sep 29 18:32:00 dist-test-slave-qcpv krb5kdc[30026](info): commencing operation
krb5kdc: starting...
W20260929 18:32:01.823473 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:32:01 dist-test-slave-qcpv krb5kdc[30026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:32:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:32:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.207.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
I20260929 18:32:01.967110 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:42219
--webserver_interface=127.28.207.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:42271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:42219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:02.102033 30039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:02.102397 30039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:02.102443 30039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:02.106424 30039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:32:02.106493 30039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:02.106514 30039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:02.106530 30039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:32:02.106547 30039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:32:02.111752 30039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:42271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:42219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:42219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:02.112857 30039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:02.113873 30039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:02.123809 30047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:02.123886 30044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:02.124082 30039 server_base.cc:1034] running on GCE node
W20260929 18:32:02.123893 30045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:02.124706 30039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:02.125958 30039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:02.127379 30039 hybrid_clock.cc:648] HybridClock initialized: now 1790706722127340 us; error 106 us; skew 500 ppm
Sep 29 18:32:02 dist-test-slave-qcpv krb5kdc[30026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706722, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:02.130686 30039 init.cc:377] Logged in from keytab as kudu/127.28.207.126@KRBTEST.COM (short username kudu)
I20260929 18:32:02.131731 30039 webserver.cc:460] Webserver started at http://127.28.207.126:41827/ using document root <none> and password file <none>
I20260929 18:32:02.132283 30039 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:02.132355 30039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:02.132527 30039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:02.136065 30039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "b8802c66d5d94c0aa6aa82c3b7fe8b80"
format_stamp: "Formatted at 2026-09-29 18:32:02 on dist-test-slave-qcpv"
server_key: "0db69abc43b1ada3757276cb5491e8cf"
server_key_iv: "0c690c1151a89b705a2a5f898b5101d9"
server_key_version: "encryptionkey@0"
I20260929 18:32:02.136925 30039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "b8802c66d5d94c0aa6aa82c3b7fe8b80"
format_stamp: "Formatted at 2026-09-29 18:32:02 on dist-test-slave-qcpv"
server_key: "0db69abc43b1ada3757276cb5491e8cf"
server_key_iv: "0c690c1151a89b705a2a5f898b5101d9"
server_key_version: "encryptionkey@0"
I20260929 18:32:02.143031 30039 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 18:32:02.146571 30054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:02.148159 30039 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:32:02.148353 30039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "b8802c66d5d94c0aa6aa82c3b7fe8b80"
format_stamp: "Formatted at 2026-09-29 18:32:02 on dist-test-slave-qcpv"
server_key: "0db69abc43b1ada3757276cb5491e8cf"
server_key_iv: "0c690c1151a89b705a2a5f898b5101d9"
server_key_version: "encryptionkey@0"
I20260929 18:32:02.148506 30039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:02.160944 30039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:02.165089 30039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:02.165355 30039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:02.173782 30039 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:42219
I20260929 18:32:02.173913 30106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:42219 every 8 connection(s)
I20260929 18:32:02.175180 30039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:32:02.176517 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 30039
I20260929 18:32:02.176683 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:32:02.176983 29501 external_mini_cluster.cc:1428] Setting key 279cb096699b87895f585ce17ebbc2e5
I20260929 18:32:02.179876 30107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 18:32:02 dist-test-slave-qcpv krb5kdc[30026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:32:02.189477 30107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8802c66d5d94c0aa6aa82c3b7fe8b80: Bootstrap starting.
I20260929 18:32:02.192566 30107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8802c66d5d94c0aa6aa82c3b7fe8b80: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:02.193647 30107 log.cc:824] T 00000000000000000000000000000000 P b8802c66d5d94c0aa6aa82c3b7fe8b80: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:02.194555 30110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:02.178941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49102 (local address 127.28.207.126:42219)
0929 18:32:02.179469 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:02.179486 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:02.179527 (+ 41us) server_negotiation.cc:407] Connection header received
0929 18:32:02.180328 (+ 801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:02.180347 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:02.180722 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:02.181043 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:02.182198 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:02.183332 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:02.184610 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:02.185103 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:02.185235 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:02.189261 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:02.189267 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:02.189281 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:02.189313 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:02.191375 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:02.191881 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:02.191886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:02.191891 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:02.192006 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:02.192301 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:02.192305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:02.192307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:02.192784 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:02.193009 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:02.193333 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 18:32:02.193669 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":165,"threads_started":1}
I20260929 18:32:02.196686 30107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8802c66d5d94c0aa6aa82c3b7fe8b80: No bootstrap required, opened a new log
I20260929 18:32:02.201449 30107 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8802c66d5d94c0aa6aa82c3b7fe8b80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8802c66d5d94c0aa6aa82c3b7fe8b80" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 42219 } }
I20260929 18:32:02.201850 30107 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8802c66d5d94c0aa6aa82c3b7fe8b80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:02.201947 30107 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8802c66d5d94c0aa6aa82c3b7fe8b80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8802c66d5d94c0aa6aa82c3b7fe8b80, State: Initialized, Role: FOLLOWER
I20260929 18:32:02.202534 30107 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8802c66d5d94c0aa6aa82c3b7fe8b80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8802c66d5d94c0aa6aa82c3b7fe8b80" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 42219 } }
I20260929 18:32:02.202723 30107 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8802c66d5d94c0aa6aa82c3b7fe8b80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:32:02.202827 30107 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8802c66d5d94c0aa6aa82c3b7fe8b80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:32:02.202967 30107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8802c66d5d94c0aa6aa82c3b7fe8b80 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:02.204449 30107 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8802c66d5d94c0aa6aa82c3b7fe8b80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8802c66d5d94c0aa6aa82c3b7fe8b80" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 42219 } }
I20260929 18:32:02.204916 30107 leader_election.cc:302] T 00000000000000000000000000000000 P b8802c66d5d94c0aa6aa82c3b7fe8b80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8802c66d5d94c0aa6aa82c3b7fe8b80; no voters:
I20260929 18:32:02.205328 30107 leader_election.cc:288] T 00000000000000000000000000000000 P b8802c66d5d94c0aa6aa82c3b7fe8b80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:32:02.205577 30112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8802c66d5d94c0aa6aa82c3b7fe8b80 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:32:02.205971 30112 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8802c66d5d94c0aa6aa82c3b7fe8b80 [term 1 LEADER]: Becoming Leader. State: Replica: b8802c66d5d94c0aa6aa82c3b7fe8b80, State: Running, Role: LEADER
I20260929 18:32:02.206388 30112 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8802c66d5d94c0aa6aa82c3b7fe8b80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8802c66d5d94c0aa6aa82c3b7fe8b80" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 42219 } }
I20260929 18:32:02.207101 30107 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8802c66d5d94c0aa6aa82c3b7fe8b80 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:32:02.208585 30114 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8802c66d5d94c0aa6aa82c3b7fe8b80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8802c66d5d94c0aa6aa82c3b7fe8b80. Latest consensus state: current_term: 1 leader_uuid: "b8802c66d5d94c0aa6aa82c3b7fe8b80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8802c66d5d94c0aa6aa82c3b7fe8b80" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 42219 } } }
I20260929 18:32:02.208738 30114 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8802c66d5d94c0aa6aa82c3b7fe8b80 [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:02.209065 30113 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8802c66d5d94c0aa6aa82c3b7fe8b80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8802c66d5d94c0aa6aa82c3b7fe8b80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8802c66d5d94c0aa6aa82c3b7fe8b80" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 42219 } } }
I20260929 18:32:02.209177 30113 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8802c66d5d94c0aa6aa82c3b7fe8b80 [sys.catalog]: This master's current role is: LEADER
W20260929 18:32:02.212357 30127 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b8802c66d5d94c0aa6aa82c3b7fe8b80: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:32:02.212461 30127 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:32:02.212554 30121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:32:02.213325 30121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:32:02.220088 30121 catalog_manager.cc:1347] Generated new cluster ID: c0af5be519da4b65bb0b639f122eedea
I20260929 18:32:02.220172 30121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:32:02.242393 30121 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:32:02.243537 30121 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:32:02.254264 30121 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8802c66d5d94c0aa6aa82c3b7fe8b80: Generated new TSK 0
I20260929 18:32:02.255276 30121 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.207.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab.
Entry for principal kudu/127.28.207.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab.
I20260929 18:32:02.299021 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.65:0
--local_ip_for_outbound_sockets=127.28.207.65
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:42219
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:42271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:02.420491 30133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:02.420791 30133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:02.420837 30133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:02.425325 30133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:02.425401 30133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:02.425465 30133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.65
I20260929 18:32:02.431090 30133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:42271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:42219
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.65
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:02.432225 30133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:02.433279 30133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:02.444339 30138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:02.444299 30141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:02.444509 30139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:02.444862 30133 server_base.cc:1034] running on GCE node
I20260929 18:32:02.445459 30133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:02.446391 30133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:02.447643 30133 hybrid_clock.cc:648] HybridClock initialized: now 1790706722447585 us; error 76 us; skew 500 ppm
Sep 29 18:32:02 dist-test-slave-qcpv krb5kdc[30026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706722, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:02.451485 30133 init.cc:377] Logged in from keytab as kudu/127.28.207.65@KRBTEST.COM (short username kudu)
I20260929 18:32:02.452766 30133 webserver.cc:460] Webserver started at http://127.28.207.65:35275/ using document root <none> and password file <none>
I20260929 18:32:02.453563 30133 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:02.453662 30133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:02.453936 30133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:02.456590 30133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-0/data/instance:
uuid: "60e9ff3abe4f42009533dea96850cf5e"
format_stamp: "Formatted at 2026-09-29 18:32:02 on dist-test-slave-qcpv"
server_key: "aa75f4304fe24139cf68d4be64715fde"
server_key_iv: "3b9c20775e3ed2d77e61b1b01dbdbc1c"
server_key_version: "encryptionkey@0"
I20260929 18:32:02.457289 30133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance:
uuid: "60e9ff3abe4f42009533dea96850cf5e"
format_stamp: "Formatted at 2026-09-29 18:32:02 on dist-test-slave-qcpv"
server_key: "aa75f4304fe24139cf68d4be64715fde"
server_key_iv: "3b9c20775e3ed2d77e61b1b01dbdbc1c"
server_key_version: "encryptionkey@0"
I20260929 18:32:02.462291 30133 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 18:32:02.466008 30148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:02.467783 30133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:32:02.467949 30133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-0/wal
uuid: "60e9ff3abe4f42009533dea96850cf5e"
format_stamp: "Formatted at 2026-09-29 18:32:02 on dist-test-slave-qcpv"
server_key: "aa75f4304fe24139cf68d4be64715fde"
server_key_iv: "3b9c20775e3ed2d77e61b1b01dbdbc1c"
server_key_version: "encryptionkey@0"
I20260929 18:32:02.468093 30133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:02.489959 30133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:02.493986 30133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:02.494259 30133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:02.495118 30133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:32:02.496208 30133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:02.496280 30133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:02.496364 30133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:02.496415 30133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:02.509809 30133 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.65:40363
I20260929 18:32:02.509846 30261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.65:40363 every 8 connection(s)
I20260929 18:32:02.511132 30133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
I20260929 18:32:02.516547 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 30133
I20260929 18:32:02.516731 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance
I20260929 18:32:02.517069 29501 external_mini_cluster.cc:1428] Setting key 805fde1a65c86b13e542fe944e5b75f4
Sep 29 18:32:02 dist-test-slave-qcpv krb5kdc[30026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706722, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.207.66@KRBTEST.COM; defaulting to no policy
I20260929 18:32:02.529707 30110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:02.513662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:33063 (local address 127.28.207.126:42219)
0929 18:32:02.513809 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:02.513812 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:02.515271 (+ 1459us) server_negotiation.cc:407] Connection header received
0929 18:32:02.516297 (+ 1026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:02.516303 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:02.516384 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:02.516494 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:02.518012 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:02.519023 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:02.521075 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:02.521524 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:02.521713 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:02.524803 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:02.524810 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:02.524817 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:02.524874 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:02.526962 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:02.527598 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:02.527603 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:02.527607 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:02.527721 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:02.528174 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:02.528179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:02.528181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:02.528507 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:02.528650 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:02.529342 (+ 692us) server_negotiation.cc:299] Negotiation successful
0929 18:32:02.529541 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 18:32:02.530612 30264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:02.514161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:42219 (local address 127.28.207.65:33063)
0929 18:32:02.515088 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:02.515142 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:02.515983 (+ 841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:02.516707 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:02.516715 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:02.517161 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:02.517753 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:02.517771 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:02.519238 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:02.519242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:02.520866 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:02.520881 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:02.521798 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:02.521804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:02.522046 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:02.522920 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:02.522948 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:02.524548 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:02.527111 (+ 2563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:02.527118 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:02.527133 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:02.527461 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:02.527857 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:02.527860 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:02.527862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:02.528034 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:02.528683 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:02.528689 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:02.528999 (+ 310us) client_negotiation.cc:769] Sending connection context
0929 18:32:02.529259 (+ 260us) client_negotiation.cc:240] Negotiation successful
0929 18:32:02.529573 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":166,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.66@KRBTEST.COM" created.
I20260929 18:32:02.532717 30262 heartbeater.cc:347] Connected to a master server at 127.28.207.126:42219
I20260929 18:32:02.533021 30262 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:02.533576 30262 heartbeater.cc:511] Master 127.28.207.126:42219 requested a full tablet report, sending...
I20260929 18:32:02.535481 30067 ts_manager.cc:194] Registered new tserver with Master: 60e9ff3abe4f42009533dea96850cf5e (127.28.207.65:40363)
I20260929 18:32:02.536741 30067 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.65@KRBTEST.COM'} at 127.28.207.65:33063
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab.
Entry for principal kudu/127.28.207.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab.
I20260929 18:32:02.555958 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.66:0
--local_ip_for_outbound_sockets=127.28.207.66
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:42219
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:42271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:02.702293 30267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:02.702688 30267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:02.702740 30267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:02.707834 30267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:02.708022 30267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:02.708122 30267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.66
I20260929 18:32:02.714668 30267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:42271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:42219
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.66
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:02.716178 30267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:02.717705 30267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:02.731304 30275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:02.731380 30267 server_base.cc:1034] running on GCE node
W20260929 18:32:02.731304 30272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:02.731305 30273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:02.732425 30267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:02.733179 30267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:02.734375 30267 hybrid_clock.cc:648] HybridClock initialized: now 1790706722734347 us; error 43 us; skew 500 ppm
Sep 29 18:32:02 dist-test-slave-qcpv krb5kdc[30026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706722, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:02.737848 30267 init.cc:377] Logged in from keytab as kudu/127.28.207.66@KRBTEST.COM (short username kudu)
I20260929 18:32:02.738971 30267 webserver.cc:460] Webserver started at http://127.28.207.66:38243/ using document root <none> and password file <none>
I20260929 18:32:02.739641 30267 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:02.739740 30267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:02.739991 30267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:02.742543 30267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-1/data/instance:
uuid: "43a2db1bd3c24c7caa6f66803cf6696a"
format_stamp: "Formatted at 2026-09-29 18:32:02 on dist-test-slave-qcpv"
server_key: "db2b86abc4753bfa3b9af8ccfa682023"
server_key_iv: "9c2160e6243ba74ff9fa00bcf501cab9"
server_key_version: "encryptionkey@0"
I20260929 18:32:02.743229 30267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance:
uuid: "43a2db1bd3c24c7caa6f66803cf6696a"
format_stamp: "Formatted at 2026-09-29 18:32:02 on dist-test-slave-qcpv"
server_key: "db2b86abc4753bfa3b9af8ccfa682023"
server_key_iv: "9c2160e6243ba74ff9fa00bcf501cab9"
server_key_version: "encryptionkey@0"
I20260929 18:32:02.747982 30267 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 18:32:02.750983 30282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:02.752552 30267 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 18:32:02.752699 30267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-1/wal
uuid: "43a2db1bd3c24c7caa6f66803cf6696a"
format_stamp: "Formatted at 2026-09-29 18:32:02 on dist-test-slave-qcpv"
server_key: "db2b86abc4753bfa3b9af8ccfa682023"
server_key_iv: "9c2160e6243ba74ff9fa00bcf501cab9"
server_key_version: "encryptionkey@0"
I20260929 18:32:02.752830 30267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:02.779686 30267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:02.783416 30267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:02.783650 30267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:02.784308 30267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:32:02.785223 30267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:02.785297 30267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:02.785364 30267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:02.785415 30267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:02.797474 30267 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.66:36569
I20260929 18:32:02.797590 30395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.66:36569 every 8 connection(s)
I20260929 18:32:02.798635 30267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
I20260929 18:32:02.805769 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 30267
I20260929 18:32:02.806027 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance
I20260929 18:32:02.806587 29501 external_mini_cluster.cc:1428] Setting key f101ac81ee5f11d011b0d2e6d0420a09
Sep 29 18:32:02 dist-test-slave-qcpv krb5kdc[30026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706722, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:32:02.818212 30110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:02.801430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:41489 (local address 127.28.207.126:42219)
0929 18:32:02.801606 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:02.801610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:02.802630 (+ 1020us) server_negotiation.cc:407] Connection header received
0929 18:32:02.804216 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:02.804219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:02.804282 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:02.804372 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:02.806694 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:02.807569 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:02.809738 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:02.810006 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:02.810150 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:02.814051 (+ 3901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:02.814055 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:02.814058 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:02.814089 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:02.815783 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:02.816454 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:02.816458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:02.816460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:02.816556 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:02.816952 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:02.816960 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:02.816962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:02.817210 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:02.817329 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:02.817880 (+ 551us) server_negotiation.cc:299] Negotiation successful
0929 18:32:02.818031 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
WARNING: no policy specified for kudu/127.28.207.67@KRBTEST.COM; defaulting to no policy
I20260929 18:32:02.819309 30398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:02.801741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:42219 (local address 127.28.207.66:41489)
0929 18:32:02.802370 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:02.802430 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:02.803911 (+ 1481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:02.804682 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:02.804693 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:02.805219 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:02.806380 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:02.806405 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:02.807798 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:02.807802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:02.809525 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:02.809541 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:02.810211 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:02.810216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:02.810481 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:02.811366 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:02.811395 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:02.813809 (+ 2414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:02.815958 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:02.815966 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:02.815989 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:02.816333 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:02.816656 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:02.816660 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:02.816662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:02.816828 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:02.817326 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:02.817334 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:02.817630 (+ 296us) client_negotiation.cc:769] Sending connection context
0929 18:32:02.817837 (+ 207us) client_negotiation.cc:240] Negotiation successful
0929 18:32:02.818232 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":139,"threads_started":1}
I20260929 18:32:02.821033 30396 heartbeater.cc:347] Connected to a master server at 127.28.207.126:42219
I20260929 18:32:02.821578 30396 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:02.822641 30396 heartbeater.cc:511] Master 127.28.207.126:42219 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.67@KRBTEST.COM" created.
I20260929 18:32:02.824230 30067 ts_manager.cc:194] Registered new tserver with Master: 43a2db1bd3c24c7caa6f66803cf6696a (127.28.207.66:36569)
I20260929 18:32:02.824887 30067 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.66@KRBTEST.COM'} at 127.28.207.66:41489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab.
Entry for principal kudu/127.28.207.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab.
I20260929 18:32:02.848834 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.67:0
--local_ip_for_outbound_sockets=127.28.207.67
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:42219
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:42271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:03.000514 30401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:03.001179 30401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:03.001554 30401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:03.007375 30401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:03.007500 30401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:03.007640 30401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.67
I20260929 18:32:03.014407 30401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:42271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:42219
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.67
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:03.016110 30401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:03.018172 30401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:03.031517 30406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:03.031513 30407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:03.031531 30409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:03.032092 30401 server_base.cc:1034] running on GCE node
I20260929 18:32:03.032615 30401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:03.033357 30401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:03.034580 30401 hybrid_clock.cc:648] HybridClock initialized: now 1790706723034552 us; error 44 us; skew 500 ppm
Sep 29 18:32:03 dist-test-slave-qcpv krb5kdc[30026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706723, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:03.037906 30401 init.cc:377] Logged in from keytab as kudu/127.28.207.67@KRBTEST.COM (short username kudu)
I20260929 18:32:03.039067 30401 webserver.cc:460] Webserver started at http://127.28.207.67:38355/ using document root <none> and password file <none>
I20260929 18:32:03.039752 30401 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:03.039835 30401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:03.040082 30401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:03.042456 30401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-2/data/instance:
uuid: "ab6f054a87b24897b772f2f25db2638d"
format_stamp: "Formatted at 2026-09-29 18:32:03 on dist-test-slave-qcpv"
server_key: "b25c160aec39e4defb3193fb3bd4eeec"
server_key_iv: "bef0e5ca28fb22f9b8ee73ff1915a593"
server_key_version: "encryptionkey@0"
I20260929 18:32:03.043123 30401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance:
uuid: "ab6f054a87b24897b772f2f25db2638d"
format_stamp: "Formatted at 2026-09-29 18:32:03 on dist-test-slave-qcpv"
server_key: "b25c160aec39e4defb3193fb3bd4eeec"
server_key_iv: "bef0e5ca28fb22f9b8ee73ff1915a593"
server_key_version: "encryptionkey@0"
I20260929 18:32:03.048028 30401 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 18:32:03.051434 30416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:03.053584 30401 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 18:32:03.053740 30401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-2/wal
uuid: "ab6f054a87b24897b772f2f25db2638d"
format_stamp: "Formatted at 2026-09-29 18:32:03 on dist-test-slave-qcpv"
server_key: "b25c160aec39e4defb3193fb3bd4eeec"
server_key_iv: "bef0e5ca28fb22f9b8ee73ff1915a593"
server_key_version: "encryptionkey@0"
I20260929 18:32:03.053881 30401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:03.099380 30401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:03.103401 30401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:03.103575 30401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:03.104166 30401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:32:03.105084 30401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:03.105168 30401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:03.105242 30401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:03.105296 30401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:03.116221 30401 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.67:39279
I20260929 18:32:03.116299 30529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.67:39279 every 8 connection(s)
I20260929 18:32:03.117208 30401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
I20260929 18:32:03.119499 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 30401
I20260929 18:32:03.119663 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance
I20260929 18:32:03.119993 29501 external_mini_cluster.cc:1428] Setting key 98763c20c613cef4d11bb9d111fec4c6
Sep 29 18:32:03 dist-test-slave-qcpv krb5kdc[30026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706723, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:32:03.136862 30110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:03.119689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:57983 (local address 127.28.207.126:42219)
0929 18:32:03.119883 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:03.119889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:03.121694 (+ 1805us) server_negotiation.cc:407] Connection header received
0929 18:32:03.123348 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:03.123351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:03.123403 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:03.123505 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:03.125512 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:03.126065 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:03.127804 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:03.128049 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:03.128153 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:03.132158 (+ 4005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:03.132162 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:03.132165 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:03.132203 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:03.134093 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:03.134931 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:03.134936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:03.134938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:03.135050 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:03.135490 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:03.135493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:03.135494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:03.135774 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:03.135871 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:03.136516 (+ 645us) server_negotiation.cc:299] Negotiation successful
0929 18:32:03.136706 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 18:32:03.137732 30532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:03.120312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:42219 (local address 127.28.207.67:57983)
0929 18:32:03.121487 (+ 1175us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:03.121568 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:03.123066 (+ 1498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:03.123861 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:03.123870 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:03.124641 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:03.125321 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:03.125336 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:03.126227 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:03.126230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:03.127592 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:03.127603 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:03.128311 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:03.128317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:03.128550 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:03.129503 (+ 953us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:03.129527 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:03.131881 (+ 2354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:03.134234 (+ 2353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:03.134241 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:03.134268 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:03.134694 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:03.135197 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:03.135200 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:03.135201 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:03.135359 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:03.135903 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:03.135909 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:03.136187 (+ 278us) client_negotiation.cc:769] Sending connection context
0929 18:32:03.136377 (+ 190us) client_negotiation.cc:240] Negotiation successful
0929 18:32:03.136684 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"thread_start_us":345,"threads_started":1}
I20260929 18:32:03.139540 30530 heartbeater.cc:347] Connected to a master server at 127.28.207.126:42219
I20260929 18:32:03.139988 30530 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:03.140772 30530 heartbeater.cc:511] Master 127.28.207.126:42219 requested a full tablet report, sending...
I20260929 18:32:03.142325 30067 ts_manager.cc:194] Registered new tserver with Master: ab6f054a87b24897b772f2f25db2638d (127.28.207.67:39279)
I20260929 18:32:03.143121 30067 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.67@KRBTEST.COM'} at 127.28.207.67:57983
I20260929 18:32:03.146642 29501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:32:03.160216 30110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:03.150628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49110 (local address 127.28.207.126:42219)
0929 18:32:03.150824 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:03.150828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:03.151098 (+ 270us) server_negotiation.cc:407] Connection header received
0929 18:32:03.151312 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:03.151321 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:03.151375 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:03.151471 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:03.152538 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:03.153142 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:03.154012 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:03.154237 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:03.154370 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:03.155895 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:03.155899 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:03.155902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:03.155937 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:03.157556 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:03.158415 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:03.158421 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:03.158423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:03.158572 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:03.159005 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:03.159008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:03.159010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:03.159323 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:03.159505 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:03.159804 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 18:32:03.159987 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 18:32:03.166549 30067 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49110:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:32:03.171224 30067 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:32:03.192646 30544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:03.185031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:39279 (local address 127.0.0.1:58610)
0929 18:32:03.185862 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:03.185873 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:03.185946 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:03.187224 (+ 1278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:03.187227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:03.187242 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:03.187329 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:03.187334 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:03.189404 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:03.189409 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:03.190587 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:03.190599 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:03.192043 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:03.192047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:03.192162 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:03.192200 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 18:32:03.192354 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 18:32:03.192497 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"mutex_wait_us":23,"thread_start_us":81,"threads_started":1}
I20260929 18:32:03.192655 30545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:03.185519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58610 (local address 127.28.207.67:39279)
0929 18:32:03.186647 (+ 1128us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:03.186657 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:03.186677 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:32:03.186743 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:03.186748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:03.186929 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:03.187069 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:03.187443 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:03.189233 (+ 1790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:03.191054 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:03.191910 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:03.192045 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:03.192206 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:03.192385 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 18:32:03.192486 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1037,"thread_start_us":91,"threads_started":1}
I20260929 18:32:03.193080 30541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:03.184240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:40363 (local address 127.0.0.1:56236)
0929 18:32:03.185226 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:03.185280 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:03.185413 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:03.186540 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:03.186543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:03.186562 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:03.186781 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:03.186806 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:03.188975 (+ 2169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:03.188978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:03.191302 (+ 2324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:03.191318 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:03.192580 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:03.192583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:03.192680 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:03.192706 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 18:32:03.192764 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 18:32:03.192868 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"mutex_wait_us":204,"thread_start_us":108,"threads_started":1}
I20260929 18:32:03.193501 30543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:03.184710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42534 (local address 127.28.207.66:36569)
0929 18:32:03.185380 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:03.185393 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:03.185808 (+ 415us) server_negotiation.cc:407] Connection header received
0929 18:32:03.185886 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:03.185891 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:03.186098 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:03.186238 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:03.186592 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:03.188766 (+ 2174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:03.191495 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:03.192405 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:03.192534 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:03.192711 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:03.193022 (+ 311us) server_negotiation.cc:299] Negotiation successful
0929 18:32:03.193273 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":142,"threads_started":1}
I20260929 18:32:03.193961 30542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:03.184510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:36569 (local address 127.0.0.1:42534)
0929 18:32:03.185636 (+ 1126us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:03.185649 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:03.185723 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:03.186266 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:03.186272 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:03.186294 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:03.186460 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:03.186471 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:03.188964 (+ 2493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:03.188970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:03.191306 (+ 2336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:03.191318 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:03.192774 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:03.192776 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:03.192861 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:03.192879 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 18:32:03.192953 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 18:32:03.193033 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1070,"mutex_wait_us":307,"thread_start_us":105,"threads_started":1}
I20260929 18:32:03.194327 30546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:03.185539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56236 (local address 127.28.207.65:40363)
0929 18:32:03.186108 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:03.186112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:03.186128 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:32:03.186221 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:03.186228 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:03.186394 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:03.187153 (+ 759us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:03.187367 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:03.188764 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:03.191484 (+ 2720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:03.192405 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:03.193757 (+ 1352us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:03.193983 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:03.194060 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 18:32:03.194179 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":103,"threads_started":1}
I20260929 18:32:03.196894 30464 tablet_service.cc:1467] Processing CreateTablet for tablet 6b19429388ea40fbb95f9118ccdc89ee (DEFAULT_TABLE table=test-table [id=aeab011945bf48c7be2a373cb224808a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:03.197268 30330 tablet_service.cc:1467] Processing CreateTablet for tablet 6b19429388ea40fbb95f9118ccdc89ee (DEFAULT_TABLE table=test-table [id=aeab011945bf48c7be2a373cb224808a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:03.198055 30464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b19429388ea40fbb95f9118ccdc89ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:03.198483 30330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b19429388ea40fbb95f9118ccdc89ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:03.201170 30196 tablet_service.cc:1467] Processing CreateTablet for tablet 6b19429388ea40fbb95f9118ccdc89ee (DEFAULT_TABLE table=test-table [id=aeab011945bf48c7be2a373cb224808a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:03.202369 30196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b19429388ea40fbb95f9118ccdc89ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:03.207625 30547 tablet_bootstrap.cc:492] T 6b19429388ea40fbb95f9118ccdc89ee P ab6f054a87b24897b772f2f25db2638d: Bootstrap starting.
I20260929 18:32:03.208034 30548 tablet_bootstrap.cc:492] T 6b19429388ea40fbb95f9118ccdc89ee P 43a2db1bd3c24c7caa6f66803cf6696a: Bootstrap starting.
I20260929 18:32:03.210219 30549 tablet_bootstrap.cc:492] T 6b19429388ea40fbb95f9118ccdc89ee P 60e9ff3abe4f42009533dea96850cf5e: Bootstrap starting.
I20260929 18:32:03.210920 30547 tablet_bootstrap.cc:654] T 6b19429388ea40fbb95f9118ccdc89ee P ab6f054a87b24897b772f2f25db2638d: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:03.211390 30548 tablet_bootstrap.cc:654] T 6b19429388ea40fbb95f9118ccdc89ee P 43a2db1bd3c24c7caa6f66803cf6696a: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:03.212487 30549 tablet_bootstrap.cc:654] T 6b19429388ea40fbb95f9118ccdc89ee P 60e9ff3abe4f42009533dea96850cf5e: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:03.212533 30547 log.cc:824] T 6b19429388ea40fbb95f9118ccdc89ee P ab6f054a87b24897b772f2f25db2638d: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:03.212533 30548 log.cc:824] T 6b19429388ea40fbb95f9118ccdc89ee P 43a2db1bd3c24c7caa6f66803cf6696a: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:03.213348 30549 log.cc:824] T 6b19429388ea40fbb95f9118ccdc89ee P 60e9ff3abe4f42009533dea96850cf5e: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:03.215224 30549 tablet_bootstrap.cc:492] T 6b19429388ea40fbb95f9118ccdc89ee P 60e9ff3abe4f42009533dea96850cf5e: No bootstrap required, opened a new log
I20260929 18:32:03.215210 30548 tablet_bootstrap.cc:492] T 6b19429388ea40fbb95f9118ccdc89ee P 43a2db1bd3c24c7caa6f66803cf6696a: No bootstrap required, opened a new log
I20260929 18:32:03.215428 30549 ts_tablet_manager.cc:1397] T 6b19429388ea40fbb95f9118ccdc89ee P 60e9ff3abe4f42009533dea96850cf5e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 18:32:03.215437 30548 ts_tablet_manager.cc:1397] T 6b19429388ea40fbb95f9118ccdc89ee P 43a2db1bd3c24c7caa6f66803cf6696a: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 18:32:03.215706 30547 tablet_bootstrap.cc:492] T 6b19429388ea40fbb95f9118ccdc89ee P ab6f054a87b24897b772f2f25db2638d: No bootstrap required, opened a new log
I20260929 18:32:03.215952 30547 ts_tablet_manager.cc:1397] T 6b19429388ea40fbb95f9118ccdc89ee P ab6f054a87b24897b772f2f25db2638d: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 18:32:03.220042 30548 raft_consensus.cc:348] T 6b19429388ea40fbb95f9118ccdc89ee P 43a2db1bd3c24c7caa6f66803cf6696a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e9ff3abe4f42009533dea96850cf5e" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 40363 } } peers { permanent_uuid: "ab6f054a87b24897b772f2f25db2638d" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 39279 } } peers { permanent_uuid: "43a2db1bd3c24c7caa6f66803cf6696a" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 36569 } }
I20260929 18:32:03.220435 30548 raft_consensus.cc:374] T 6b19429388ea40fbb95f9118ccdc89ee P 43a2db1bd3c24c7caa6f66803cf6696a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:03.220489 30548 raft_consensus.cc:729] T 6b19429388ea40fbb95f9118ccdc89ee P 43a2db1bd3c24c7caa6f66803cf6696a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43a2db1bd3c24c7caa6f66803cf6696a, State: Initialized, Role: FOLLOWER
I20260929 18:32:03.220225 30547 raft_consensus.cc:348] T 6b19429388ea40fbb95f9118ccdc89ee P ab6f054a87b24897b772f2f25db2638d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e9ff3abe4f42009533dea96850cf5e" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 40363 } } peers { permanent_uuid: "ab6f054a87b24897b772f2f25db2638d" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 39279 } } peers { permanent_uuid: "43a2db1bd3c24c7caa6f66803cf6696a" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 36569 } }
I20260929 18:32:03.220577 30547 raft_consensus.cc:374] T 6b19429388ea40fbb95f9118ccdc89ee P ab6f054a87b24897b772f2f25db2638d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:03.220402 30549 raft_consensus.cc:348] T 6b19429388ea40fbb95f9118ccdc89ee P 60e9ff3abe4f42009533dea96850cf5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e9ff3abe4f42009533dea96850cf5e" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 40363 } } peers { permanent_uuid: "ab6f054a87b24897b772f2f25db2638d" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 39279 } } peers { permanent_uuid: "43a2db1bd3c24c7caa6f66803cf6696a" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 36569 } }
I20260929 18:32:03.220670 30547 raft_consensus.cc:729] T 6b19429388ea40fbb95f9118ccdc89ee P ab6f054a87b24897b772f2f25db2638d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab6f054a87b24897b772f2f25db2638d, State: Initialized, Role: FOLLOWER
I20260929 18:32:03.220669 30549 raft_consensus.cc:374] T 6b19429388ea40fbb95f9118ccdc89ee P 60e9ff3abe4f42009533dea96850cf5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:03.220726 30549 raft_consensus.cc:729] T 6b19429388ea40fbb95f9118ccdc89ee P 60e9ff3abe4f42009533dea96850cf5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60e9ff3abe4f42009533dea96850cf5e, State: Initialized, Role: FOLLOWER
I20260929 18:32:03.221158 30548 consensus_queue.cc:260] T 6b19429388ea40fbb95f9118ccdc89ee P 43a2db1bd3c24c7caa6f66803cf6696a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e9ff3abe4f42009533dea96850cf5e" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 40363 } } peers { permanent_uuid: "ab6f054a87b24897b772f2f25db2638d" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 39279 } } peers { permanent_uuid: "43a2db1bd3c24c7caa6f66803cf6696a" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 36569 } }
I20260929 18:32:03.221216 30549 consensus_queue.cc:260] T 6b19429388ea40fbb95f9118ccdc89ee P 60e9ff3abe4f42009533dea96850cf5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e9ff3abe4f42009533dea96850cf5e" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 40363 } } peers { permanent_uuid: "ab6f054a87b24897b772f2f25db2638d" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 39279 } } peers { permanent_uuid: "43a2db1bd3c24c7caa6f66803cf6696a" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 36569 } }
I20260929 18:32:03.221293 30547 consensus_queue.cc:260] T 6b19429388ea40fbb95f9118ccdc89ee P ab6f054a87b24897b772f2f25db2638d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e9ff3abe4f42009533dea96850cf5e" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 40363 } } peers { permanent_uuid: "ab6f054a87b24897b772f2f25db2638d" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 39279 } } peers { permanent_uuid: "43a2db1bd3c24c7caa6f66803cf6696a" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 36569 } }
I20260929 18:32:03.222373 30396 heartbeater.cc:503] Master 127.28.207.126:42219 was elected leader, sending a full tablet report...
I20260929 18:32:03.222437 30549 ts_tablet_manager.cc:1428] T 6b19429388ea40fbb95f9118ccdc89ee P 60e9ff3abe4f42009533dea96850cf5e: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 18:32:03.222437 30548 ts_tablet_manager.cc:1428] T 6b19429388ea40fbb95f9118ccdc89ee P 43a2db1bd3c24c7caa6f66803cf6696a: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 18:32:03.222644 30547 ts_tablet_manager.cc:1428] T 6b19429388ea40fbb95f9118ccdc89ee P ab6f054a87b24897b772f2f25db2638d: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 18:32:03.223119 30530 heartbeater.cc:503] Master 127.28.207.126:42219 was elected leader, sending a full tablet report...
I20260929 18:32:03.223228 30262 heartbeater.cc:503] Master 127.28.207.126:42219 was elected leader, sending a full tablet report...
I20260929 18:32:03.226058 30555 raft_consensus.cc:482] T 6b19429388ea40fbb95f9118ccdc89ee P ab6f054a87b24897b772f2f25db2638d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:32:03.226271 30555 raft_consensus.cc:504] T 6b19429388ea40fbb95f9118ccdc89ee P ab6f054a87b24897b772f2f25db2638d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e9ff3abe4f42009533dea96850cf5e" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 40363 } } peers { permanent_uuid: "ab6f054a87b24897b772f2f25db2638d" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 39279 } } peers { permanent_uuid: "43a2db1bd3c24c7caa6f66803cf6696a" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 36569 } }
I20260929 18:32:03.227861 30555 leader_election.cc:288] T 6b19429388ea40fbb95f9118ccdc89ee P ab6f054a87b24897b772f2f25db2638d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60e9ff3abe4f42009533dea96850cf5e (127.28.207.65:40363), 43a2db1bd3c24c7caa6f66803cf6696a (127.28.207.66:36569)
I20260929 18:32:03.232983 30543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:03.229100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:45837 (local address 127.28.207.66:36569)
0929 18:32:03.229244 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:03.229250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:03.229265 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:32:03.229315 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:03.229318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:03.229365 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:03.229437 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:03.229838 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:03.230473 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:03.231859 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:03.232316 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:03.232463 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:03.232537 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:03.232718 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 18:32:03.232813 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 18:32:03.232998 30546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:03.228261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:54381 (local address 127.28.207.65:40363)
0929 18:32:03.228714 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:03.228721 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:03.228737 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:32:03.228792 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:03.228795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:03.228845 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:03.228916 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:03.229846 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:03.230521 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:03.231906 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:03.232336 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:03.232491 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:03.232547 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:03.232802 (+ 255us) server_negotiation.cc:299] Negotiation successful
0929 18:32:03.232886 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364}
I20260929 18:32:03.233019 30532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:03.228138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:40363 (local address 127.28.207.67:54381)
0929 18:32:03.228355 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:03.228372 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:03.228449 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:03.229532 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:03.229534 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:03.229549 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:03.229686 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:03.229693 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:03.230690 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:03.230694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:03.231723 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:03.231734 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:03.232501 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:03.232505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:03.232627 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:03.232647 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 18:32:03.232764 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 18:32:03.232852 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":159}
I20260929 18:32:03.233023 30556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:03.228449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:36569 (local address 127.28.207.67:45837)
0929 18:32:03.229039 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:03.229052 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:03.229130 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:03.229444 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:03.229447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:03.229558 (+ 111us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:03.229699 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:03.229704 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:03.230608 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:03.230615 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:03.231722 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:03.231733 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:03.232456 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:03.232459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:03.232573 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:03.232605 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 18:32:03.232672 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 18:32:03.232774 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":40,"spinlock_wait_cycles":29440,"thread_start_us":122,"threads_started":1}
I20260929 18:32:03.233801 30350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b19429388ea40fbb95f9118ccdc89ee" candidate_uuid: "ab6f054a87b24897b772f2f25db2638d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43a2db1bd3c24c7caa6f66803cf6696a" is_pre_election: true
I20260929 18:32:03.233949 30216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b19429388ea40fbb95f9118ccdc89ee" candidate_uuid: "ab6f054a87b24897b772f2f25db2638d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60e9ff3abe4f42009533dea96850cf5e" is_pre_election: true
I20260929 18:32:03.234153 30350 raft_consensus.cc:2454] T 6b19429388ea40fbb95f9118ccdc89ee P 43a2db1bd3c24c7caa6f66803cf6696a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab6f054a87b24897b772f2f25db2638d in term 0.
I20260929 18:32:03.234377 30216 raft_consensus.cc:2454] T 6b19429388ea40fbb95f9118ccdc89ee P 60e9ff3abe4f42009533dea96850cf5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab6f054a87b24897b772f2f25db2638d in term 0.
I20260929 18:32:03.234773 30420 leader_election.cc:302] T 6b19429388ea40fbb95f9118ccdc89ee P ab6f054a87b24897b772f2f25db2638d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43a2db1bd3c24c7caa6f66803cf6696a, ab6f054a87b24897b772f2f25db2638d; no voters:
I20260929 18:32:03.235121 30555 raft_consensus.cc:2781] T 6b19429388ea40fbb95f9118ccdc89ee P ab6f054a87b24897b772f2f25db2638d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:32:03.235275 30555 raft_consensus.cc:482] T 6b19429388ea40fbb95f9118ccdc89ee P ab6f054a87b24897b772f2f25db2638d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:32:03.235345 30555 raft_consensus.cc:3037] T 6b19429388ea40fbb95f9118ccdc89ee P ab6f054a87b24897b772f2f25db2638d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:03.236419 30555 raft_consensus.cc:504] T 6b19429388ea40fbb95f9118ccdc89ee P ab6f054a87b24897b772f2f25db2638d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e9ff3abe4f42009533dea96850cf5e" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 40363 } } peers { permanent_uuid: "ab6f054a87b24897b772f2f25db2638d" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 39279 } } peers { permanent_uuid: "43a2db1bd3c24c7caa6f66803cf6696a" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 36569 } }
I20260929 18:32:03.236836 30555 leader_election.cc:288] T 6b19429388ea40fbb95f9118ccdc89ee P ab6f054a87b24897b772f2f25db2638d [CANDIDATE]: Term 1 election: Requested vote from peers 60e9ff3abe4f42009533dea96850cf5e (127.28.207.65:40363), 43a2db1bd3c24c7caa6f66803cf6696a (127.28.207.66:36569)
I20260929 18:32:03.237242 30350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b19429388ea40fbb95f9118ccdc89ee" candidate_uuid: "ab6f054a87b24897b772f2f25db2638d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43a2db1bd3c24c7caa6f66803cf6696a"
I20260929 18:32:03.237303 30216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b19429388ea40fbb95f9118ccdc89ee" candidate_uuid: "ab6f054a87b24897b772f2f25db2638d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60e9ff3abe4f42009533dea96850cf5e"
I20260929 18:32:03.237351 30350 raft_consensus.cc:3037] T 6b19429388ea40fbb95f9118ccdc89ee P 43a2db1bd3c24c7caa6f66803cf6696a [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:03.237530 30216 raft_consensus.cc:3037] T 6b19429388ea40fbb95f9118ccdc89ee P 60e9ff3abe4f42009533dea96850cf5e [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:03.238539 30350 raft_consensus.cc:2454] T 6b19429388ea40fbb95f9118ccdc89ee P 43a2db1bd3c24c7caa6f66803cf6696a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab6f054a87b24897b772f2f25db2638d in term 1.
I20260929 18:32:03.238644 30216 raft_consensus.cc:2454] T 6b19429388ea40fbb95f9118ccdc89ee P 60e9ff3abe4f42009533dea96850cf5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab6f054a87b24897b772f2f25db2638d in term 1.
I20260929 18:32:03.238919 30420 leader_election.cc:302] T 6b19429388ea40fbb95f9118ccdc89ee P ab6f054a87b24897b772f2f25db2638d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43a2db1bd3c24c7caa6f66803cf6696a, ab6f054a87b24897b772f2f25db2638d; no voters:
I20260929 18:32:03.239135 30555 raft_consensus.cc:2781] T 6b19429388ea40fbb95f9118ccdc89ee P ab6f054a87b24897b772f2f25db2638d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:32:03.239426 30555 raft_consensus.cc:686] T 6b19429388ea40fbb95f9118ccdc89ee P ab6f054a87b24897b772f2f25db2638d [term 1 LEADER]: Becoming Leader. State: Replica: ab6f054a87b24897b772f2f25db2638d, State: Running, Role: LEADER
I20260929 18:32:03.239830 30555 consensus_queue.cc:237] T 6b19429388ea40fbb95f9118ccdc89ee P ab6f054a87b24897b772f2f25db2638d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e9ff3abe4f42009533dea96850cf5e" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 40363 } } peers { permanent_uuid: "ab6f054a87b24897b772f2f25db2638d" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 39279 } } peers { permanent_uuid: "43a2db1bd3c24c7caa6f66803cf6696a" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 36569 } }
I20260929 18:32:03.243404 30071 catalog_manager.cc:5495] T 6b19429388ea40fbb95f9118ccdc89ee P ab6f054a87b24897b772f2f25db2638d reported cstate change: term changed from 0 to 1, leader changed from <none> to ab6f054a87b24897b772f2f25db2638d (127.28.207.67). New cstate: current_term: 1 leader_uuid: "ab6f054a87b24897b772f2f25db2638d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e9ff3abe4f42009533dea96850cf5e" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 40363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab6f054a87b24897b772f2f25db2638d" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 39279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43a2db1bd3c24c7caa6f66803cf6696a" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 36569 } health_report { overall_health: UNKNOWN } } }
Sep 29 18:32:03 dist-test-slave-qcpv krb5kdc[30026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.207.65@KRBTEST.COM
I20260929 18:32:03.261188 30546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:03.249831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56246 (local address 127.28.207.65:40363)
0929 18:32:03.250060 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:03.250066 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:03.250281 (+ 215us) server_negotiation.cc:407] Connection header received
0929 18:32:03.250455 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:03.250459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:03.250511 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:03.250607 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:03.251416 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:03.252012 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:03.252958 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:03.253250 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:03.253366 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:03.256595 (+ 3229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:03.256607 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:03.256622 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:03.256670 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:03.258617 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:03.259353 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:03.259358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:03.259359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:03.259450 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:03.260104 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:03.260107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:03.260108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:03.260369 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:03.260531 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:03.260916 (+ 385us) server_negotiation.cc:299] Negotiation successful
0929 18:32:03.261049 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
W20260929 18:32:03.262809 30263 tablet.cc:2369] T 6b19429388ea40fbb95f9118ccdc89ee P 60e9ff3abe4f42009533dea96850cf5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 18:32:03 dist-test-slave-qcpv krb5kdc[30026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706723, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 18:32:03 dist-test-slave-qcpv krb5kdc[30026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706723, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.207.65@KRBTEST.COM
I20260929 18:32:03.292580 30546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:03.281940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56248 (local address 127.28.207.65:40363)
0929 18:32:03.282172 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:03.282177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:03.282422 (+ 245us) server_negotiation.cc:407] Connection header received
0929 18:32:03.282811 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:03.282815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:03.282928 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:03.283036 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:03.283863 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:03.284458 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:03.285559 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:03.285818 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:03.285949 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:03.288886 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:03.288890 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:03.288892 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:03.288930 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:03.290527 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:03.291125 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:03.291130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:03.291132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:03.291217 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:03.291516 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:03.291521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:03.291522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:03.291733 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:03.291883 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:03.292226 (+ 343us) server_negotiation.cc:299] Negotiation successful
0929 18:32:03.292372 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
W20260929 18:32:03.300459 30397 tablet.cc:2369] T 6b19429388ea40fbb95f9118ccdc89ee P 43a2db1bd3c24c7caa6f66803cf6696a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 18:32:03 dist-test-slave-qcpv krb5kdc[30026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 18:32:03 dist-test-slave-qcpv krb5kdc[30026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.207.65@KRBTEST.COM
I20260929 18:32:03.325191 30546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:03.314033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56250 (local address 127.28.207.65:40363)
0929 18:32:03.314304 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:03.314311 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:03.314522 (+ 211us) server_negotiation.cc:407] Connection header received
0929 18:32:03.314730 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:03.314740 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:03.314792 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:03.314963 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:03.315872 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:03.316440 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:03.317449 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:03.317732 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:03.317838 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:03.320972 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:03.320977 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:03.320979 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:03.321019 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:03.322688 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:03.323501 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:03.323504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:03.323506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:03.323605 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:03.324038 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:03.324041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:03.324042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:03.324411 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:03.324540 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:03.324885 (+ 345us) server_negotiation.cc:299] Negotiation successful
0929 18:32:03.325034 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
W20260929 18:32:03.327225 30176 tablet_service.cc:2908] Rejecting scan request for tablet 6b19429388ea40fbb95f9118ccdc89ee: Uninitialized: clean time has not yet been initialized
I20260929 18:32:03.330415 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 30133
I20260929 18:32:03.338264 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 30267
I20260929 18:32:03.345453 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 30401
I20260929 18:32:03.351881 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 30039
2026-09-29T18:32:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2598 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:32:03 dist-test-slave-qcpv krb5kdc[30571](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:32:03 dist-test-slave-qcpv krb5kdc[30571](info): set up 2 sockets
Sep 29 18:32:03 dist-test-slave-qcpv krb5kdc[30571](info): commencing operation
krb5kdc: starting...
W20260929 18:32:04.455343 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:32:04 dist-test-slave-qcpv krb5kdc[30571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:32:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:32:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.207.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
I20260929 18:32:04.599165 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:36751
--webserver_interface=127.28.207.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:33435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:36751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:04.747344 30584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:04.747788 30584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:04.747879 30584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:04.752616 30584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:32:04.752756 30584 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 18:32:04.752790 30584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:04.752835 30584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:04.752890 30584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:32:04.752928 30584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:32:04.761777 30584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:33435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:36751
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:36751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:04.763909 30584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:04.765618 30584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:04.777724 30590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:04.777685 30589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:04.777752 30592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:04.778412 30584 server_base.cc:1034] running on GCE node
I20260929 18:32:04.779194 30584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:04.780730 30584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:04.784178 30584 hybrid_clock.cc:648] HybridClock initialized: now 1790706724784031 us; error 174 us; skew 500 ppm
Sep 29 18:32:04 dist-test-slave-qcpv krb5kdc[30571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706724, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:04.787950 30584 init.cc:377] Logged in from keytab as kudu/127.28.207.126@KRBTEST.COM (short username kudu)
I20260929 18:32:04.789193 30584 webserver.cc:460] Webserver started at http://127.28.207.126:45417/ using document root <none> and password file <none>
I20260929 18:32:04.789853 30584 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:04.789959 30584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:04.790217 30584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:04.792667 30584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "7a031dfb803c416a988640504a679e57"
format_stamp: "Formatted at 2026-09-29 18:32:04 on dist-test-slave-qcpv"
server_key: "50c9a15a9a8c6bb7237b497a1f9b492f"
server_key_iv: "b33631cfa57069d0eb9fe73ead35e5d6"
server_key_version: "encryptionkey@0"
I20260929 18:32:04.793285 30584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "7a031dfb803c416a988640504a679e57"
format_stamp: "Formatted at 2026-09-29 18:32:04 on dist-test-slave-qcpv"
server_key: "50c9a15a9a8c6bb7237b497a1f9b492f"
server_key_iv: "b33631cfa57069d0eb9fe73ead35e5d6"
server_key_version: "encryptionkey@0"
I20260929 18:32:04.798034 30584 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 18:32:04.801433 30599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:04.803362 30584 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260929 18:32:04.803548 30584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "7a031dfb803c416a988640504a679e57"
format_stamp: "Formatted at 2026-09-29 18:32:04 on dist-test-slave-qcpv"
server_key: "50c9a15a9a8c6bb7237b497a1f9b492f"
server_key_iv: "b33631cfa57069d0eb9fe73ead35e5d6"
server_key_version: "encryptionkey@0"
I20260929 18:32:04.803689 30584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:04.814538 30584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:04.818002 30584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:04.818243 30584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:04.827041 30584 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:36751
I20260929 18:32:04.827106 30661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:36751 every 8 connection(s)
I20260929 18:32:04.828140 30584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:32:04.832330 30662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:04.838001 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 30584
I20260929 18:32:04.838168 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:32:04.838474 29501 external_mini_cluster.cc:1428] Setting key 7ae38b70b0a6419d0951635035b16305
I20260929 18:32:04.840823 30662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a031dfb803c416a988640504a679e57: Bootstrap starting.
I20260929 18:32:04.843899 30662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a031dfb803c416a988640504a679e57: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:04.845038 30662 log.cc:824] T 00000000000000000000000000000000 P 7a031dfb803c416a988640504a679e57: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:04.848174 30662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a031dfb803c416a988640504a679e57: No bootstrap required, opened a new log
Sep 29 18:32:04 dist-test-slave-qcpv krb5kdc[30571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:32:04.853132 30662 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a031dfb803c416a988640504a679e57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a031dfb803c416a988640504a679e57" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 36751 } }
I20260929 18:32:04.853511 30662 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a031dfb803c416a988640504a679e57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:04.853953 30662 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a031dfb803c416a988640504a679e57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a031dfb803c416a988640504a679e57, State: Initialized, Role: FOLLOWER
I20260929 18:32:04.854657 30662 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a031dfb803c416a988640504a679e57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a031dfb803c416a988640504a679e57" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 36751 } }
I20260929 18:32:04.854945 30662 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a031dfb803c416a988640504a679e57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:32:04.855064 30662 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a031dfb803c416a988640504a679e57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:32:04.855201 30662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a031dfb803c416a988640504a679e57 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:04.856446 30662 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a031dfb803c416a988640504a679e57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a031dfb803c416a988640504a679e57" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 36751 } }
I20260929 18:32:04.856936 30662 leader_election.cc:302] T 00000000000000000000000000000000 P 7a031dfb803c416a988640504a679e57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a031dfb803c416a988640504a679e57; no voters:
I20260929 18:32:04.857092 30665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:04.840455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51554 (local address 127.28.207.126:36751)
0929 18:32:04.841205 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:04.841226 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:04.841278 (+ 52us) server_negotiation.cc:407] Connection header received
0929 18:32:04.842179 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:04.842200 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:04.842630 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:04.843095 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:04.843995 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:04.845139 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:04.846270 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:04.846787 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:04.847053 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:04.850210 (+ 3157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:04.850216 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:04.850232 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:04.850262 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:04.852508 (+ 2246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:04.853059 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:04.853065 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:04.853070 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:04.853185 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:04.853477 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:04.853483 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:04.853487 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:04.854322 (+ 835us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:04.854566 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:04.855601 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0929 18:32:04.855954 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":203,"threads_started":1}
I20260929 18:32:04.857415 30662 leader_election.cc:288] T 00000000000000000000000000000000 P 7a031dfb803c416a988640504a679e57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:32:04.857527 30667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a031dfb803c416a988640504a679e57 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:32:04.857801 30667 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a031dfb803c416a988640504a679e57 [term 1 LEADER]: Becoming Leader. State: Replica: 7a031dfb803c416a988640504a679e57, State: Running, Role: LEADER
I20260929 18:32:04.858275 30667 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a031dfb803c416a988640504a679e57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a031dfb803c416a988640504a679e57" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 36751 } }
I20260929 18:32:04.859249 30662 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a031dfb803c416a988640504a679e57 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:32:04.860599 30669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a031dfb803c416a988640504a679e57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a031dfb803c416a988640504a679e57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a031dfb803c416a988640504a679e57" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 36751 } } }
I20260929 18:32:04.860560 30668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a031dfb803c416a988640504a679e57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a031dfb803c416a988640504a679e57. Latest consensus state: current_term: 1 leader_uuid: "7a031dfb803c416a988640504a679e57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a031dfb803c416a988640504a679e57" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 36751 } } }
I20260929 18:32:04.860792 30669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a031dfb803c416a988640504a679e57 [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:04.860798 30668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a031dfb803c416a988640504a679e57 [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:04.861238 30676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:32:04.865329 30676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:32:04.872747 30676 catalog_manager.cc:1347] Generated new cluster ID: e783deb8c4094083ac26a84f174768b1
I20260929 18:32:04.872864 30676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:32:04.895932 30676 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:32:04.897854 30676 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:32:04.907146 30676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a031dfb803c416a988640504a679e57: Generated new TSK 0
I20260929 18:32:04.908648 30676 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.207.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab.
Entry for principal kudu/127.28.207.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab.
I20260929 18:32:04.959470 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.65:0
--local_ip_for_outbound_sockets=127.28.207.65
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:36751
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:33435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:05.116381 30688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:05.116706 30688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:05.116757 30688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:05.121253 30688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:05.121331 30688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:05.121374 30688 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:32:05.121409 30688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.65
I20260929 18:32:05.127404 30688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:33435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.207.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.207.126:36751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.65
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:05.128763 30688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:05.129976 30688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:05.141664 30693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:05.141733 30696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:05.141733 30694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:05.142341 30688 server_base.cc:1034] running on GCE node
I20260929 18:32:05.143077 30688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:05.143893 30688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:05.145109 30688 hybrid_clock.cc:648] HybridClock initialized: now 1790706725145082 us; error 44 us; skew 500 ppm
Sep 29 18:32:05 dist-test-slave-qcpv krb5kdc[30571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706725, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:05.148931 30688 init.cc:377] Logged in from keytab as kudu/127.28.207.65@KRBTEST.COM (short username kudu)
I20260929 18:32:05.149955 30688 webserver.cc:460] Webserver started at http://127.28.207.65:46431/ using document root <none> and password file <none>
I20260929 18:32:05.150568 30688 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:05.150677 30688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:05.150990 30688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:05.153270 30688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-0/data/instance:
uuid: "28f15042e6c541e4b27b316554dbddfd"
format_stamp: "Formatted at 2026-09-29 18:32:05 on dist-test-slave-qcpv"
server_key: "6f3f05bcfc4080659f8c68a53397a7ff"
server_key_iv: "63d531b2f69618e1a4014825783ce0b8"
server_key_version: "encryptionkey@0"
I20260929 18:32:05.153892 30688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance:
uuid: "28f15042e6c541e4b27b316554dbddfd"
format_stamp: "Formatted at 2026-09-29 18:32:05 on dist-test-slave-qcpv"
server_key: "6f3f05bcfc4080659f8c68a53397a7ff"
server_key_iv: "63d531b2f69618e1a4014825783ce0b8"
server_key_version: "encryptionkey@0"
I20260929 18:32:05.158339 30688 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 18:32:05.161998 30703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:05.163993 30688 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:32:05.164219 30688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-0/wal
uuid: "28f15042e6c541e4b27b316554dbddfd"
format_stamp: "Formatted at 2026-09-29 18:32:05 on dist-test-slave-qcpv"
server_key: "6f3f05bcfc4080659f8c68a53397a7ff"
server_key_iv: "63d531b2f69618e1a4014825783ce0b8"
server_key_version: "encryptionkey@0"
I20260929 18:32:05.164381 30688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:05.197515 30688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:05.201264 30688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:05.201493 30688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:05.203168 30688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:05.203251 30688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:05.203322 30688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:05.203367 30688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:05.215477 30688 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.65:44095
I20260929 18:32:05.215588 30818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.65:44095 every 8 connection(s)
I20260929 18:32:05.216673 30688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
Sep 29 18:32:05 dist-test-slave-qcpv krb5kdc[30571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706725, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:32:05.220446 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 30688
I20260929 18:32:05.220788 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance
I20260929 18:32:05.221171 29501 external_mini_cluster.cc:1428] Setting key 45152f96d66aaa4fb5a6428f19bd8dd5
I20260929 18:32:05.223479 30665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:05.204951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:44051 (local address 127.28.207.126:36751)
0929 18:32:05.205121 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:05.205125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:05.206342 (+ 1217us) server_negotiation.cc:407] Connection header received
0929 18:32:05.208630 (+ 2288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:05.208637 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:05.208726 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:05.208840 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:05.210882 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:05.211806 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:05.214262 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:05.214667 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:05.214813 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:05.218562 (+ 3749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:05.218566 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:05.218576 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:05.218608 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:05.220394 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:05.221119 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:05.221124 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:05.221126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:05.221220 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:05.221607 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:05.221612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:05.221614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:05.221861 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:05.221972 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:05.223113 (+ 1141us) server_negotiation.cc:299] Negotiation successful
0929 18:32:05.223309 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 18:32:05.224767 30713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:05.205418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36751 (local address 127.28.207.65:44051)
0929 18:32:05.206103 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:05.206182 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:05.207714 (+ 1532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:05.209149 (+ 1435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:05.209165 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:05.209770 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:05.210629 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:05.210658 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:05.212057 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:05.212065 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:05.214064 (+ 1999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:05.214083 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:05.215075 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:05.215081 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:05.215343 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:05.216197 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:05.216218 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:05.218293 (+ 2075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:05.220556 (+ 2263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:05.220563 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:05.220581 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:05.221009 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:05.221353 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:05.221356 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:05.221358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:05.221504 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:05.222093 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:05.222105 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:05.222709 (+ 604us) client_negotiation.cc:769] Sending connection context
0929 18:32:05.223027 (+ 318us) client_negotiation.cc:240] Negotiation successful
0929 18:32:05.223525 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":130,"threads_started":1}
I20260929 18:32:05.226538 30819 heartbeater.cc:347] Connected to a master server at 127.28.207.126:36751
I20260929 18:32:05.226714 30819 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:05.227660 30819 heartbeater.cc:511] Master 127.28.207.126:36751 requested a full tablet report, sending...
I20260929 18:32:05.230631 30616 ts_manager.cc:194] Registered new tserver with Master: 28f15042e6c541e4b27b316554dbddfd (127.28.207.65:44095)
WARNING: no policy specified for kudu/127.28.207.66@KRBTEST.COM; defaulting to no policy
I20260929 18:32:05.232532 30616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.65@KRBTEST.COM'} at 127.28.207.65:44051
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.66@KRBTEST.COM" created.
I20260929 18:32:05.243016 30665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:05.233709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:41343 (local address 127.28.207.126:36751)
0929 18:32:05.233851 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:05.233855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:05.233870 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:32:05.233913 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:05.233915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:05.233954 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:05.234089 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:05.234838 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:05.235651 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:05.237055 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:05.237503 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:05.237649 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:05.239208 (+ 1559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:05.239212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:05.239215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:05.239239 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:05.241107 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:05.241617 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:05.241622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:05.241624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:05.241713 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:05.242027 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:05.242031 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:05.242033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:05.242253 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:05.242372 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:05.242654 (+ 282us) server_negotiation.cc:299] Negotiation successful
0929 18:32:05.242788 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:32:05.243044 30826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:05.230121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36751 (local address 127.28.207.65:41343)
0929 18:32:05.230524 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:05.230544 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:05.230681 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:05.234223 (+ 3542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:05.234227 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:05.234450 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:05.234619 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:05.234626 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:05.235825 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:05.235832 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:05.236881 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:05.236891 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:05.237674 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:05.237678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:05.237760 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:05.238486 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:05.238500 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:05.239017 (+ 517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:05.241257 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:05.241261 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:05.241264 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:05.241500 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:05.241823 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:05.241826 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:05.241828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:05.241924 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:05.242356 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:05.242359 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:05.242469 (+ 110us) client_negotiation.cc:769] Sending connection context
0929 18:32:05.242584 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 18:32:05.242788 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":148,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab.
Entry for principal kudu/127.28.207.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab.
I20260929 18:32:05.255234 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.66:0
--local_ip_for_outbound_sockets=127.28.207.66
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:36751
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:33435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:05.379465 30828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:05.379829 30828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:05.379909 30828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:05.384630 30828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:05.384799 30828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:05.384887 30828 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:32:05.384976 30828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.66
I20260929 18:32:05.391007 30828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:33435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.207.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.207.126:36751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.66
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:05.392268 30828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:05.393365 30828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:05.404717 30833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:05.404762 30834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:05.405041 30828 server_base.cc:1034] running on GCE node
W20260929 18:32:05.405234 30836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:05.405896 30828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:05.407014 30828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:05.408286 30828 hybrid_clock.cc:648] HybridClock initialized: now 1790706725408215 us; error 94 us; skew 500 ppm
Sep 29 18:32:05 dist-test-slave-qcpv krb5kdc[30571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706725, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:05.412657 30828 init.cc:377] Logged in from keytab as kudu/127.28.207.66@KRBTEST.COM (short username kudu)
I20260929 18:32:05.414018 30828 webserver.cc:460] Webserver started at http://127.28.207.66:36337/ using document root <none> and password file <none>
I20260929 18:32:05.414840 30828 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:05.415033 30828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:05.415341 30828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:05.418572 30828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-1/data/instance:
uuid: "3c52aae6cbe94546b373509a42d6ca54"
format_stamp: "Formatted at 2026-09-29 18:32:05 on dist-test-slave-qcpv"
server_key: "21f7b22b9c031434a979b3f41f8d9a6d"
server_key_iv: "715fcac77aa1a277605e4968cead1509"
server_key_version: "encryptionkey@0"
I20260929 18:32:05.419466 30828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance:
uuid: "3c52aae6cbe94546b373509a42d6ca54"
format_stamp: "Formatted at 2026-09-29 18:32:05 on dist-test-slave-qcpv"
server_key: "21f7b22b9c031434a979b3f41f8d9a6d"
server_key_iv: "715fcac77aa1a277605e4968cead1509"
server_key_version: "encryptionkey@0"
I20260929 18:32:05.424975 30828 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 18:32:05.429051 30843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:05.431257 30828 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 18:32:05.431465 30828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-1/wal
uuid: "3c52aae6cbe94546b373509a42d6ca54"
format_stamp: "Formatted at 2026-09-29 18:32:05 on dist-test-slave-qcpv"
server_key: "21f7b22b9c031434a979b3f41f8d9a6d"
server_key_iv: "715fcac77aa1a277605e4968cead1509"
server_key_version: "encryptionkey@0"
I20260929 18:32:05.431636 30828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:05.446537 30828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:05.451622 30828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:05.452035 30828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:05.454273 30828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:05.454386 30828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:05.454466 30828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:05.454514 30828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:32:05 dist-test-slave-qcpv krb5kdc[30571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706725, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:32:05.469406 30828 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.66:38011
I20260929 18:32:05.469456 30958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.66:38011 every 8 connection(s)
I20260929 18:32:05.471004 30828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
I20260929 18:32:05.473222 30852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:05.455520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36751 (local address 127.28.207.66:60759)
0929 18:32:05.456075 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:05.456139 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:05.457283 (+ 1144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:05.458455 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:05.458468 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:05.459156 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:05.460041 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:05.460064 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:05.461636 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:05.461640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:05.462998 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:05.463011 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:05.463688 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:05.463692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:05.463897 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:05.464630 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:05.464673 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:05.466941 (+ 2268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:05.469561 (+ 2620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:05.469568 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:05.469586 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:05.469898 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:05.470330 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:05.470333 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:05.470335 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:05.470482 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:05.471181 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:05.471189 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:05.471598 (+ 409us) client_negotiation.cc:769] Sending connection context
0929 18:32:05.471854 (+ 256us) client_negotiation.cc:240] Negotiation successful
0929 18:32:05.472249 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":167,"threads_started":1}
I20260929 18:32:05.473974 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 30828
I20260929 18:32:05.474107 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance
I20260929 18:32:05.474175 30665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:05.454962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:60759 (local address 127.28.207.126:36751)
0929 18:32:05.455189 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:05.455194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:05.456310 (+ 1116us) server_negotiation.cc:407] Connection header received
0929 18:32:05.457579 (+ 1269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:05.457584 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:05.457953 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:05.458093 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:05.460266 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:05.461412 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:05.463147 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:05.463485 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:05.463609 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:05.467151 (+ 3542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:05.467157 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:05.467165 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:05.467192 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:05.469391 (+ 2199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:05.470051 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:05.470055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:05.470058 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:05.470180 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:05.470622 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:05.470626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:05.470627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:05.471001 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:05.471112 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:05.473823 (+ 2711us) server_negotiation.cc:299] Negotiation successful
0929 18:32:05.473991 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 18:32:05.474465 29501 external_mini_cluster.cc:1428] Setting key 0bdd9801b6293e1e835399de35a7b047
I20260929 18:32:05.475745 30959 heartbeater.cc:347] Connected to a master server at 127.28.207.126:36751
I20260929 18:32:05.475975 30959 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:05.476722 30959 heartbeater.cc:511] Master 127.28.207.126:36751 requested a full tablet report, sending...
I20260929 18:32:05.479418 30612 ts_manager.cc:194] Registered new tserver with Master: 3c52aae6cbe94546b373509a42d6ca54 (127.28.207.66:38011)
I20260929 18:32:05.480621 30612 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.66@KRBTEST.COM'} at 127.28.207.66:60759
WARNING: no policy specified for kudu/127.28.207.67@KRBTEST.COM; defaulting to no policy
I20260929 18:32:05.489162 30665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:05.479394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:56963 (local address 127.28.207.126:36751)
0929 18:32:05.479573 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:05.479581 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:05.479953 (+ 372us) server_negotiation.cc:407] Connection header received
0929 18:32:05.480100 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:05.480106 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:05.480182 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:05.480301 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:05.480934 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:05.481676 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:05.482748 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:05.483085 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:05.483206 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:05.484836 (+ 1630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:05.484840 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:05.484843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:05.484878 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:05.487024 (+ 2146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:05.487633 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:05.487637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:05.487638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:05.487763 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:05.488134 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:05.488137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:05.488138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:05.488383 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:05.488472 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:05.488894 (+ 422us) server_negotiation.cc:299] Negotiation successful
0929 18:32:05.489017 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":71}
I20260929 18:32:05.489177 30966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:05.479276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36751 (local address 127.28.207.66:56963)
0929 18:32:05.479812 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:05.479831 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:05.479974 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:05.480341 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:05.480346 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:05.480613 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:05.480786 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:05.480796 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:05.481814 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:05.481818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:05.482621 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:05.482629 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:05.483215 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:05.483219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:05.483319 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:05.484224 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:05.484237 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:05.484657 (+ 420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:05.487203 (+ 2546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:05.487207 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:05.487210 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:05.487485 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:05.487906 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:05.487909 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:05.487911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:05.488024 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:05.488559 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:05.488565 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:05.488720 (+ 155us) client_negotiation.cc:769] Sending connection context
0929 18:32:05.488822 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 18:32:05.488979 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":30,"thread_start_us":131,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab.
Entry for principal kudu/127.28.207.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab.
I20260929 18:32:05.511579 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.67:0
--local_ip_for_outbound_sockets=127.28.207.67
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:36751
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:33435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:05.635819 30968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:05.636209 30968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:05.636266 30968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:05.641010 30968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:05.641112 30968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:05.641157 30968 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:32:05.641193 30968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.67
I20260929 18:32:05.647615 30968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:33435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.207.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.207.126:36751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.67
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:05.648847 30968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:05.649940 30968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:05.662132 30973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:05.662132 30974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:05.662348 30976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:05.662902 30968 server_base.cc:1034] running on GCE node
I20260929 18:32:05.663709 30968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:05.664714 30968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:05.665983 30968 hybrid_clock.cc:648] HybridClock initialized: now 1790706725665938 us; error 79 us; skew 500 ppm
Sep 29 18:32:05 dist-test-slave-qcpv krb5kdc[30571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706725, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:05.670431 30968 init.cc:377] Logged in from keytab as kudu/127.28.207.67@KRBTEST.COM (short username kudu)
I20260929 18:32:05.671691 30968 webserver.cc:460] Webserver started at http://127.28.207.67:35745/ using document root <none> and password file <none>
I20260929 18:32:05.672346 30968 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:05.672418 30968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:05.672657 30968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:05.675221 30968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-2/data/instance:
uuid: "a4642f7baed54c09994b275f9837d000"
format_stamp: "Formatted at 2026-09-29 18:32:05 on dist-test-slave-qcpv"
server_key: "d9dfb6a290ba2918bef6be04afda83c3"
server_key_iv: "8e3b6a1f90a6fda935f71997ce565de4"
server_key_version: "encryptionkey@0"
I20260929 18:32:05.675984 30968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance:
uuid: "a4642f7baed54c09994b275f9837d000"
format_stamp: "Formatted at 2026-09-29 18:32:05 on dist-test-slave-qcpv"
server_key: "d9dfb6a290ba2918bef6be04afda83c3"
server_key_iv: "8e3b6a1f90a6fda935f71997ce565de4"
server_key_version: "encryptionkey@0"
I20260929 18:32:05.681147 30968 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 18:32:05.685676 30983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:05.687621 30968 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 18:32:05.687789 30968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-2/wal
uuid: "a4642f7baed54c09994b275f9837d000"
format_stamp: "Formatted at 2026-09-29 18:32:05 on dist-test-slave-qcpv"
server_key: "d9dfb6a290ba2918bef6be04afda83c3"
server_key_iv: "8e3b6a1f90a6fda935f71997ce565de4"
server_key_version: "encryptionkey@0"
I20260929 18:32:05.687904 30968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:05.701246 30968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:05.705116 30968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:05.705289 30968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:05.706947 30968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:05.707026 30968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:05.707090 30968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:05.707175 30968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:32:05 dist-test-slave-qcpv krb5kdc[30571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706725, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:32:05.720813 30968 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.67:37409
I20260929 18:32:05.720906 31098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.67:37409 every 8 connection(s)
I20260929 18:32:05.722101 30968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
I20260929 18:32:05.725589 30665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:05.708364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:37381 (local address 127.28.207.126:36751)
0929 18:32:05.708565 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:05.708571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:05.709837 (+ 1266us) server_negotiation.cc:407] Connection header received
0929 18:32:05.711453 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:05.711461 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:05.711550 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:05.711701 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:05.713594 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:05.714547 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:05.716672 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:05.717033 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:05.717230 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:05.721237 (+ 4007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:05.721241 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:05.721244 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:05.721275 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:05.723260 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:05.723838 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:05.723841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:05.723843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:05.723932 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:05.724293 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:05.724296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:05.724298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:05.724525 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:05.724636 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:05.725288 (+ 652us) server_negotiation.cc:299] Negotiation successful
0929 18:32:05.725419 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 18:32:05.726658 30993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:05.709058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36751 (local address 127.28.207.67:37381)
0929 18:32:05.709644 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:05.709703 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:05.711157 (+ 1454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:05.711958 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:05.711966 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:05.712683 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:05.713358 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:05.713377 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:05.714757 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:05.714764 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:05.716474 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:05.716491 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:05.717296 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:05.717300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:05.717506 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:05.718432 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:05.718466 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:05.720993 (+ 2527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:05.723411 (+ 2418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:05.723418 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:05.723436 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:05.723714 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:05.724030 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:05.724034 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:05.724036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:05.724186 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:05.724644 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:05.724653 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:05.724923 (+ 270us) client_negotiation.cc:769] Sending connection context
0929 18:32:05.725222 (+ 299us) client_negotiation.cc:240] Negotiation successful
0929 18:32:05.725579 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":182,"threads_started":1}
I20260929 18:32:05.728235 31099 heartbeater.cc:347] Connected to a master server at 127.28.207.126:36751
I20260929 18:32:05.728394 31099 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:05.729003 31099 heartbeater.cc:511] Master 127.28.207.126:36751 requested a full tablet report, sending...
I20260929 18:32:05.730433 30616 ts_manager.cc:194] Registered new tserver with Master: a4642f7baed54c09994b275f9837d000 (127.28.207.67:37409)
I20260929 18:32:05.731380 30616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.67@KRBTEST.COM'} at 127.28.207.67:37381
I20260929 18:32:05.731643 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 30968
I20260929 18:32:05.731797 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance
I20260929 18:32:05.732198 29501 external_mini_cluster.cc:1428] Setting key f3f59c88ba90033294dc942e85f0a9e9
I20260929 18:32:05.735078 29501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:32:05.741888 31105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:05.731380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36751 (local address 127.28.207.67:39687)
0929 18:32:05.731882 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:05.731902 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:05.732051 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:05.732470 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:05.732474 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:05.732744 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:05.732921 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:05.732932 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:05.734102 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:05.734105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:05.734691 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:05.734699 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:05.735486 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:05.735490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:05.735609 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:05.736389 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:05.736410 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:05.736919 (+ 509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:05.739887 (+ 2968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:05.739891 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:05.739893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:05.740193 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:05.740615 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:05.740617 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:05.740619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:05.740716 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:05.741357 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:05.741359 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:05.741471 (+ 112us) client_negotiation.cc:769] Sending connection context
0929 18:32:05.741580 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 18:32:05.741724 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":4,"thread_start_us":155,"threads_started":1}
I20260929 18:32:05.742306 30665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:05.731441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:39687 (local address 127.28.207.126:36751)
0929 18:32:05.731568 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:05.731573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:05.732011 (+ 438us) server_negotiation.cc:407] Connection header received
0929 18:32:05.732170 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:05.732173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:05.732224 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:05.732327 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:05.733053 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:05.733779 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:05.734834 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:05.735226 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:05.735328 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:05.737195 (+ 1867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:05.737203 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:05.737210 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:05.737259 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:05.739590 (+ 2331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:05.740333 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:05.740339 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:05.740341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:05.740464 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:05.740840 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:05.740846 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:05.740851 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:05.741185 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:05.741320 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:05.741697 (+ 377us) server_negotiation.cc:299] Negotiation successful
0929 18:32:05.741882 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 18:32:05 dist-test-slave-qcpv krb5kdc[30571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706725, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 18:32:05 dist-test-slave-qcpv krb5kdc[30571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706725, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:32:05.770223 30665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:05.759413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41726 (local address 127.28.207.126:36751)
0929 18:32:05.759767 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:05.759777 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:05.759808 (+ 31us) server_negotiation.cc:407] Connection header received
0929 18:32:05.759865 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:05.759868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:05.759921 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:05.760039 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:05.760967 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:05.761606 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:05.762512 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:05.762736 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:05.762891 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:05.766027 (+ 3136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:05.766033 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:05.766035 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:05.766078 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:05.767739 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:05.768427 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:05.768429 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:05.768431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:05.768539 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:05.769129 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:05.769131 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:05.769133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:05.769389 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:05.769539 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:05.769909 (+ 370us) server_negotiation.cc:299] Negotiation successful
0929 18:32:05.770033 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260929 18:32:05.773906 30616 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:32:05.777529 30616 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:32:05.796137 31121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:05.789633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34022 (local address 127.28.207.66:38011)
0929 18:32:05.790394 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:05.790403 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:05.790421 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:32:05.790487 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:05.790491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:05.790662 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:05.791094 (+ 432us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:05.791340 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:05.792665 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:05.794475 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:05.795275 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:05.795400 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:05.795569 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:05.795828 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 18:32:05.795940 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":102,"threads_started":1}
I20260929 18:32:05.796216 31120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:05.789473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:38011 (local address 127.0.0.1:34022)
0929 18:32:05.789844 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:05.789908 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:05.790108 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:05.790832 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:05.790834 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:05.790863 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:05.791199 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:05.791209 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:05.792785 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:05.792788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:05.794327 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:05.794340 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:05.795435 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:05.795439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:05.795576 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:05.795621 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 18:32:05.795766 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 18:32:05.795941 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":140,"spinlock_wait_cycles":80896,"thread_start_us":97,"threads_started":1}
I20260929 18:32:05.796562 31118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:05.788659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50434 (local address 127.28.207.65:44095)
0929 18:32:05.789570 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:05.789582 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:05.790265 (+ 683us) server_negotiation.cc:407] Connection header received
0929 18:32:05.790365 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:05.790371 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:05.790628 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:05.790801 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:05.791273 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:05.792473 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:05.794350 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:05.795275 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:05.795388 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:05.795551 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:05.796287 (+ 736us) server_negotiation.cc:299] Negotiation successful
0929 18:32:05.796397 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":766,"thread_start_us":126,"threads_started":1}
I20260929 18:32:05.796221 31116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:05.788518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:44095 (local address 127.0.0.1:50434)
0929 18:32:05.789992 (+ 1474us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:05.790007 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:05.790105 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:05.790825 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:05.790831 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:05.790896 (+ 65us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:05.791118 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:05.791128 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:05.792608 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:05.792612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:05.794212 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:05.794223 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:05.795625 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:05.795629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:05.795724 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:05.795795 (+ 71us) client_negotiation.cc:769] Sending connection context
0929 18:32:05.795873 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 18:32:05.795984 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"mutex_wait_us":234,"spinlock_wait_cycles":17280,"thread_start_us":92,"threads_started":1}
I20260929 18:32:05.797809 31119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:05.788816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41792 (local address 127.28.207.67:37409)
0929 18:32:05.789202 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:05.789215 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:05.790266 (+ 1051us) server_negotiation.cc:407] Connection header received
0929 18:32:05.790365 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:05.790371 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:05.790628 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:05.790801 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:05.791433 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:05.792871 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:05.794643 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:05.795425 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:05.795545 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:05.795768 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:05.797490 (+ 1722us) server_negotiation.cc:299] Negotiation successful
0929 18:32:05.797625 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":127,"threads_started":1}
I20260929 18:32:05.798115 31117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:05.788518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:37409 (local address 127.0.0.1:41792)
0929 18:32:05.789842 (+ 1324us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:05.789910 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:05.790105 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:05.791014 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:05.791019 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:05.791044 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:05.791301 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:05.791309 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:05.793286 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:05.793290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:05.794521 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:05.794528 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:05.795829 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:05.795833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:05.795921 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:05.795941 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 18:32:05.796009 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 18:32:05.796092 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1145,"mutex_wait_us":220,"thread_start_us":138,"threads_started":1}
I20260929 18:32:05.800025 30747 tablet_service.cc:1467] Processing CreateTablet for tablet 75358c021ebd4641ac3de3556179faab (DEFAULT_TABLE table=test-table [id=ed05f75b28364f3d9ed9c6d064d9d07e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:05.801083 30747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75358c021ebd4641ac3de3556179faab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:05.800209 30893 tablet_service.cc:1467] Processing CreateTablet for tablet 75358c021ebd4641ac3de3556179faab (DEFAULT_TABLE table=test-table [id=ed05f75b28364f3d9ed9c6d064d9d07e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:05.801994 30893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75358c021ebd4641ac3de3556179faab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:05.802060 31033 tablet_service.cc:1467] Processing CreateTablet for tablet 75358c021ebd4641ac3de3556179faab (DEFAULT_TABLE table=test-table [id=ed05f75b28364f3d9ed9c6d064d9d07e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:05.803157 31033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75358c021ebd4641ac3de3556179faab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:05.808676 31122 tablet_bootstrap.cc:492] T 75358c021ebd4641ac3de3556179faab P 28f15042e6c541e4b27b316554dbddfd: Bootstrap starting.
I20260929 18:32:05.810966 31122 tablet_bootstrap.cc:654] T 75358c021ebd4641ac3de3556179faab P 28f15042e6c541e4b27b316554dbddfd: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:05.811785 31122 log.cc:824] T 75358c021ebd4641ac3de3556179faab P 28f15042e6c541e4b27b316554dbddfd: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:05.813699 31124 tablet_bootstrap.cc:492] T 75358c021ebd4641ac3de3556179faab P 3c52aae6cbe94546b373509a42d6ca54: Bootstrap starting.
I20260929 18:32:05.814172 31122 tablet_bootstrap.cc:492] T 75358c021ebd4641ac3de3556179faab P 28f15042e6c541e4b27b316554dbddfd: No bootstrap required, opened a new log
I20260929 18:32:05.814357 31125 tablet_bootstrap.cc:492] T 75358c021ebd4641ac3de3556179faab P a4642f7baed54c09994b275f9837d000: Bootstrap starting.
I20260929 18:32:05.814410 31122 ts_tablet_manager.cc:1397] T 75358c021ebd4641ac3de3556179faab P 28f15042e6c541e4b27b316554dbddfd: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 18:32:05.817019 31125 tablet_bootstrap.cc:654] T 75358c021ebd4641ac3de3556179faab P a4642f7baed54c09994b275f9837d000: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:05.817978 31125 log.cc:824] T 75358c021ebd4641ac3de3556179faab P a4642f7baed54c09994b275f9837d000: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:05.819600 31122 raft_consensus.cc:348] T 75358c021ebd4641ac3de3556179faab P 28f15042e6c541e4b27b316554dbddfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 } } peers { permanent_uuid: "a4642f7baed54c09994b275f9837d000" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37409 } } peers { permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 } }
I20260929 18:32:05.819904 31122 raft_consensus.cc:374] T 75358c021ebd4641ac3de3556179faab P 28f15042e6c541e4b27b316554dbddfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:05.819962 31122 raft_consensus.cc:729] T 75358c021ebd4641ac3de3556179faab P 28f15042e6c541e4b27b316554dbddfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28f15042e6c541e4b27b316554dbddfd, State: Initialized, Role: FOLLOWER
I20260929 18:32:05.820355 31124 tablet_bootstrap.cc:654] T 75358c021ebd4641ac3de3556179faab P 3c52aae6cbe94546b373509a42d6ca54: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:05.820539 31125 tablet_bootstrap.cc:492] T 75358c021ebd4641ac3de3556179faab P a4642f7baed54c09994b275f9837d000: No bootstrap required, opened a new log
I20260929 18:32:05.820452 31122 consensus_queue.cc:260] T 75358c021ebd4641ac3de3556179faab P 28f15042e6c541e4b27b316554dbddfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 } } peers { permanent_uuid: "a4642f7baed54c09994b275f9837d000" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37409 } } peers { permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 } }
I20260929 18:32:05.820748 31125 ts_tablet_manager.cc:1397] T 75358c021ebd4641ac3de3556179faab P a4642f7baed54c09994b275f9837d000: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 18:32:05.821384 30819 heartbeater.cc:503] Master 127.28.207.126:36751 was elected leader, sending a full tablet report...
I20260929 18:32:05.821470 31124 log.cc:824] T 75358c021ebd4641ac3de3556179faab P 3c52aae6cbe94546b373509a42d6ca54: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:05.821543 31122 ts_tablet_manager.cc:1428] T 75358c021ebd4641ac3de3556179faab P 28f15042e6c541e4b27b316554dbddfd: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 18:32:05.824730 31124 tablet_bootstrap.cc:492] T 75358c021ebd4641ac3de3556179faab P 3c52aae6cbe94546b373509a42d6ca54: No bootstrap required, opened a new log
I20260929 18:32:05.824936 31124 ts_tablet_manager.cc:1397] T 75358c021ebd4641ac3de3556179faab P 3c52aae6cbe94546b373509a42d6ca54: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260929 18:32:05.825012 31125 raft_consensus.cc:348] T 75358c021ebd4641ac3de3556179faab P a4642f7baed54c09994b275f9837d000 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 } } peers { permanent_uuid: "a4642f7baed54c09994b275f9837d000" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37409 } } peers { permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 } }
I20260929 18:32:05.825299 31125 raft_consensus.cc:374] T 75358c021ebd4641ac3de3556179faab P a4642f7baed54c09994b275f9837d000 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:05.825366 31125 raft_consensus.cc:729] T 75358c021ebd4641ac3de3556179faab P a4642f7baed54c09994b275f9837d000 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4642f7baed54c09994b275f9837d000, State: Initialized, Role: FOLLOWER
I20260929 18:32:05.825845 31125 consensus_queue.cc:260] T 75358c021ebd4641ac3de3556179faab P a4642f7baed54c09994b275f9837d000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 } } peers { permanent_uuid: "a4642f7baed54c09994b275f9837d000" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37409 } } peers { permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 } }
I20260929 18:32:05.827328 31099 heartbeater.cc:503] Master 127.28.207.126:36751 was elected leader, sending a full tablet report...
I20260929 18:32:05.828783 31125 ts_tablet_manager.cc:1428] T 75358c021ebd4641ac3de3556179faab P a4642f7baed54c09994b275f9837d000: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260929 18:32:05.831074 31124 raft_consensus.cc:348] T 75358c021ebd4641ac3de3556179faab P 3c52aae6cbe94546b373509a42d6ca54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 } } peers { permanent_uuid: "a4642f7baed54c09994b275f9837d000" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37409 } } peers { permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 } }
I20260929 18:32:05.831558 31124 raft_consensus.cc:374] T 75358c021ebd4641ac3de3556179faab P 3c52aae6cbe94546b373509a42d6ca54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:05.831635 31124 raft_consensus.cc:729] T 75358c021ebd4641ac3de3556179faab P 3c52aae6cbe94546b373509a42d6ca54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c52aae6cbe94546b373509a42d6ca54, State: Initialized, Role: FOLLOWER
I20260929 18:32:05.832768 31124 consensus_queue.cc:260] T 75358c021ebd4641ac3de3556179faab P 3c52aae6cbe94546b373509a42d6ca54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 } } peers { permanent_uuid: "a4642f7baed54c09994b275f9837d000" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37409 } } peers { permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 } }
I20260929 18:32:05.833704 30959 heartbeater.cc:503] Master 127.28.207.126:36751 was elected leader, sending a full tablet report...
I20260929 18:32:05.834199 31124 ts_tablet_manager.cc:1428] T 75358c021ebd4641ac3de3556179faab P 3c52aae6cbe94546b373509a42d6ca54: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
W20260929 18:32:05.968456 30820 tablet.cc:2369] T 75358c021ebd4641ac3de3556179faab P 28f15042e6c541e4b27b316554dbddfd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:32:05.972879 30960 tablet.cc:2369] T 75358c021ebd4641ac3de3556179faab P 3c52aae6cbe94546b373509a42d6ca54: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:32:05.973519 31100 tablet.cc:2369] T 75358c021ebd4641ac3de3556179faab P a4642f7baed54c09994b275f9837d000: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:32:06.054242 31127 raft_consensus.cc:482] T 75358c021ebd4641ac3de3556179faab P 28f15042e6c541e4b27b316554dbddfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:32:06.054659 31127 raft_consensus.cc:504] T 75358c021ebd4641ac3de3556179faab P 28f15042e6c541e4b27b316554dbddfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 } } peers { permanent_uuid: "a4642f7baed54c09994b275f9837d000" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37409 } } peers { permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 } }
I20260929 18:32:06.055198 31129 raft_consensus.cc:482] T 75358c021ebd4641ac3de3556179faab P a4642f7baed54c09994b275f9837d000 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:32:06.055496 31129 raft_consensus.cc:504] T 75358c021ebd4641ac3de3556179faab P a4642f7baed54c09994b275f9837d000 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 } } peers { permanent_uuid: "a4642f7baed54c09994b275f9837d000" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37409 } } peers { permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 } }
I20260929 18:32:06.056471 31127 leader_election.cc:288] T 75358c021ebd4641ac3de3556179faab P 28f15042e6c541e4b27b316554dbddfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4642f7baed54c09994b275f9837d000 (127.28.207.67:37409), 3c52aae6cbe94546b373509a42d6ca54 (127.28.207.66:38011)
I20260929 18:32:06.058187 31129 leader_election.cc:288] T 75358c021ebd4641ac3de3556179faab P a4642f7baed54c09994b275f9837d000 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28f15042e6c541e4b27b316554dbddfd (127.28.207.65:44095), 3c52aae6cbe94546b373509a42d6ca54 (127.28.207.66:38011)
I20260929 18:32:06.062793 31119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:06.057018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:47957 (local address 127.28.207.67:37409)
0929 18:32:06.057188 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:06.057193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:06.057503 (+ 310us) server_negotiation.cc:407] Connection header received
0929 18:32:06.057794 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:06.057798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:06.057859 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:06.057916 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:06.058441 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.059534 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.061364 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.061987 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.062155 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:06.062224 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:06.062504 (+ 280us) server_negotiation.cc:299] Negotiation successful
0929 18:32:06.062621 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":64}
I20260929 18:32:06.063081 31121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:06.057788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:47993 (local address 127.28.207.66:38011)
0929 18:32:06.058022 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:06.058028 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:06.058094 (+ 66us) server_negotiation.cc:407] Connection header received
0929 18:32:06.058318 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:06.058321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:06.058387 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:06.058440 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:06.059241 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.060001 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.061469 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.062210 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.062329 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:06.062432 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:06.062764 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 18:32:06.062893 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":98,"server-negotiator.queue_time_us":153}
I20260929 18:32:06.063302 31134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:06.058049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:39213 (local address 127.28.207.66:38011)
0929 18:32:06.058434 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:06.058441 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:06.058457 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:32:06.058526 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:06.058529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:06.058584 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:06.058753 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:06.059215 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.060121 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.061679 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.062408 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.062918 (+ 510us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:06.062992 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:06.063038 (+ 46us) server_negotiation.cc:299] Negotiation successful
0929 18:32:06.063133 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":327,"thread_start_us":58,"threads_started":1}
I20260929 18:32:06.063594 30993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:06.057208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:38011 (local address 127.28.207.67:39213)
0929 18:32:06.057426 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:06.057444 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:06.057590 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:06.058709 (+ 1119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:06.058712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:06.058724 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:06.059081 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:06.059091 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.060244 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.060247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:06.061511 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:06.061521 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.062538 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.062542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:06.062642 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:06.062664 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 18:32:06.062709 (+ 45us) client_negotiation.cc:240] Negotiation successful
0929 18:32:06.062775 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":186}
I20260929 18:32:06.063894 31132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:06.057389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:38011 (local address 127.28.207.65:47993)
0929 18:32:06.057994 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:06.058008 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:06.058180 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:06.058582 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:06.058584 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:06.058800 (+ 216us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:06.059085 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:06.059092 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.060175 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.060180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:06.061352 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:06.061359 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.062488 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.062491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:06.062607 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:06.062633 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 18:32:06.063352 (+ 719us) client_negotiation.cc:240] Negotiation successful
0929 18:32:06.063442 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":79,"spinlock_wait_cycles":5632,"thread_start_us":119,"threads_started":1}
I20260929 18:32:06.064045 31118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:06.057768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:58297 (local address 127.28.207.65:44095)
0929 18:32:06.057968 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:06.057974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:06.058469 (+ 495us) server_negotiation.cc:407] Connection header received
0929 18:32:06.058690 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:06.058694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:06.058818 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:06.058918 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:06.059911 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.060640 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.062413 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.063016 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.063111 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:06.063179 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:06.063794 (+ 615us) server_negotiation.cc:299] Negotiation successful
0929 18:32:06.063888 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":94,"spinlock_wait_cycles":130816}
I20260929 18:32:06.064519 31133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:06.057208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:44095 (local address 127.28.207.67:58297)
0929 18:32:06.058231 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:06.058245 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:06.058551 (+ 306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:06.059102 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:06.059105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:06.059123 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:06.059777 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:06.059787 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.060839 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.060842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:06.062286 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:06.062293 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.063289 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.063292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:06.063407 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:06.063430 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:32:06.063922 (+ 492us) client_negotiation.cc:240] Negotiation successful
0929 18:32:06.064344 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":941,"mutex_wait_us":701,"thread_start_us":117,"threads_started":1}
I20260929 18:32:06.064813 31131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:06.056881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:37409 (local address 127.28.207.65:47957)
0929 18:32:06.057388 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:06.057407 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:06.057604 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:06.058101 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:06.058104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:06.058123 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:06.058317 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:06.058325 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.059716 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.059720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:06.061216 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:06.061227 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.062194 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.062198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:06.062316 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:06.062345 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 18:32:06.063513 (+ 1168us) client_negotiation.cc:240] Negotiation successful
0929 18:32:06.063630 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":173,"threads_started":1}
I20260929 18:32:06.064792 30913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75358c021ebd4641ac3de3556179faab" candidate_uuid: "a4642f7baed54c09994b275f9837d000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c52aae6cbe94546b373509a42d6ca54" is_pre_election: true
I20260929 18:32:06.064808 30912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75358c021ebd4641ac3de3556179faab" candidate_uuid: "28f15042e6c541e4b27b316554dbddfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c52aae6cbe94546b373509a42d6ca54" is_pre_election: true
I20260929 18:32:06.065270 30913 raft_consensus.cc:2454] T 75358c021ebd4641ac3de3556179faab P 3c52aae6cbe94546b373509a42d6ca54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4642f7baed54c09994b275f9837d000 in term 0.
I20260929 18:32:06.065173 30772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75358c021ebd4641ac3de3556179faab" candidate_uuid: "a4642f7baed54c09994b275f9837d000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28f15042e6c541e4b27b316554dbddfd" is_pre_election: true
I20260929 18:32:06.065582 30772 raft_consensus.cc:2454] T 75358c021ebd4641ac3de3556179faab P 28f15042e6c541e4b27b316554dbddfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4642f7baed54c09994b275f9837d000 in term 0.
I20260929 18:32:06.065968 31053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75358c021ebd4641ac3de3556179faab" candidate_uuid: "28f15042e6c541e4b27b316554dbddfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4642f7baed54c09994b275f9837d000" is_pre_election: true
I20260929 18:32:06.066068 30985 leader_election.cc:302] T 75358c021ebd4641ac3de3556179faab P a4642f7baed54c09994b275f9837d000 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 28f15042e6c541e4b27b316554dbddfd, 3c52aae6cbe94546b373509a42d6ca54, a4642f7baed54c09994b275f9837d000; no voters:
I20260929 18:32:06.066212 31053 raft_consensus.cc:2454] T 75358c021ebd4641ac3de3556179faab P a4642f7baed54c09994b275f9837d000 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28f15042e6c541e4b27b316554dbddfd in term 0.
I20260929 18:32:06.066422 31129 raft_consensus.cc:2781] T 75358c021ebd4641ac3de3556179faab P a4642f7baed54c09994b275f9837d000 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:32:06.066563 31129 raft_consensus.cc:482] T 75358c021ebd4641ac3de3556179faab P a4642f7baed54c09994b275f9837d000 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:32:06.066735 31129 raft_consensus.cc:3037] T 75358c021ebd4641ac3de3556179faab P a4642f7baed54c09994b275f9837d000 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:06.066812 30705 leader_election.cc:302] T 75358c021ebd4641ac3de3556179faab P 28f15042e6c541e4b27b316554dbddfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 28f15042e6c541e4b27b316554dbddfd, a4642f7baed54c09994b275f9837d000; no voters: 3c52aae6cbe94546b373509a42d6ca54
I20260929 18:32:06.067210 31127 raft_consensus.cc:2781] T 75358c021ebd4641ac3de3556179faab P 28f15042e6c541e4b27b316554dbddfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:32:06.067349 31127 raft_consensus.cc:482] T 75358c021ebd4641ac3de3556179faab P 28f15042e6c541e4b27b316554dbddfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:32:06.067417 31127 raft_consensus.cc:3037] T 75358c021ebd4641ac3de3556179faab P 28f15042e6c541e4b27b316554dbddfd [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:06.068454 31129 raft_consensus.cc:504] T 75358c021ebd4641ac3de3556179faab P a4642f7baed54c09994b275f9837d000 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 } } peers { permanent_uuid: "a4642f7baed54c09994b275f9837d000" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37409 } } peers { permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 } }
I20260929 18:32:06.068735 31127 raft_consensus.cc:504] T 75358c021ebd4641ac3de3556179faab P 28f15042e6c541e4b27b316554dbddfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 } } peers { permanent_uuid: "a4642f7baed54c09994b275f9837d000" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37409 } } peers { permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 } }
I20260929 18:32:06.068954 31129 leader_election.cc:288] T 75358c021ebd4641ac3de3556179faab P a4642f7baed54c09994b275f9837d000 [CANDIDATE]: Term 1 election: Requested vote from peers 28f15042e6c541e4b27b316554dbddfd (127.28.207.65:44095), 3c52aae6cbe94546b373509a42d6ca54 (127.28.207.66:38011)
I20260929 18:32:06.069216 31127 leader_election.cc:288] T 75358c021ebd4641ac3de3556179faab P 28f15042e6c541e4b27b316554dbddfd [CANDIDATE]: Term 1 election: Requested vote from peers a4642f7baed54c09994b275f9837d000 (127.28.207.67:37409), 3c52aae6cbe94546b373509a42d6ca54 (127.28.207.66:38011)
I20260929 18:32:06.069399 30772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75358c021ebd4641ac3de3556179faab" candidate_uuid: "a4642f7baed54c09994b275f9837d000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28f15042e6c541e4b27b316554dbddfd"
I20260929 18:32:06.069428 30912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75358c021ebd4641ac3de3556179faab" candidate_uuid: "a4642f7baed54c09994b275f9837d000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c52aae6cbe94546b373509a42d6ca54"
I20260929 18:32:06.069640 30772 raft_consensus.cc:2379] T 75358c021ebd4641ac3de3556179faab P 28f15042e6c541e4b27b316554dbddfd [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a4642f7baed54c09994b275f9837d000 in current term 1: Already voted for candidate 28f15042e6c541e4b27b316554dbddfd in this term.
I20260929 18:32:06.069567 30912 raft_consensus.cc:3037] T 75358c021ebd4641ac3de3556179faab P 3c52aae6cbe94546b373509a42d6ca54 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:06.070484 30913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75358c021ebd4641ac3de3556179faab" candidate_uuid: "28f15042e6c541e4b27b316554dbddfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c52aae6cbe94546b373509a42d6ca54"
I20260929 18:32:06.070582 31053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75358c021ebd4641ac3de3556179faab" candidate_uuid: "28f15042e6c541e4b27b316554dbddfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4642f7baed54c09994b275f9837d000"
I20260929 18:32:06.070753 31053 raft_consensus.cc:2379] T 75358c021ebd4641ac3de3556179faab P a4642f7baed54c09994b275f9837d000 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 28f15042e6c541e4b27b316554dbddfd in current term 1: Already voted for candidate a4642f7baed54c09994b275f9837d000 in this term.
I20260929 18:32:06.071224 30705 leader_election.cc:302] T 75358c021ebd4641ac3de3556179faab P 28f15042e6c541e4b27b316554dbddfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28f15042e6c541e4b27b316554dbddfd; no voters: 3c52aae6cbe94546b373509a42d6ca54, a4642f7baed54c09994b275f9837d000
I20260929 18:32:06.071295 30912 raft_consensus.cc:2454] T 75358c021ebd4641ac3de3556179faab P 3c52aae6cbe94546b373509a42d6ca54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4642f7baed54c09994b275f9837d000 in term 1.
I20260929 18:32:06.071444 31127 raft_consensus.cc:2726] T 75358c021ebd4641ac3de3556179faab P 28f15042e6c541e4b27b316554dbddfd [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260929 18:32:06.071807 30985 leader_election.cc:302] T 75358c021ebd4641ac3de3556179faab P a4642f7baed54c09994b275f9837d000 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c52aae6cbe94546b373509a42d6ca54, a4642f7baed54c09994b275f9837d000; no voters: 28f15042e6c541e4b27b316554dbddfd
I20260929 18:32:06.072158 31129 raft_consensus.cc:2781] T 75358c021ebd4641ac3de3556179faab P a4642f7baed54c09994b275f9837d000 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:32:06.072623 31129 raft_consensus.cc:686] T 75358c021ebd4641ac3de3556179faab P a4642f7baed54c09994b275f9837d000 [term 1 LEADER]: Becoming Leader. State: Replica: a4642f7baed54c09994b275f9837d000, State: Running, Role: LEADER
I20260929 18:32:06.073092 31129 consensus_queue.cc:237] T 75358c021ebd4641ac3de3556179faab P a4642f7baed54c09994b275f9837d000 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 } } peers { permanent_uuid: "a4642f7baed54c09994b275f9837d000" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37409 } } peers { permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 } }
I20260929 18:32:06.077308 30616 catalog_manager.cc:5495] T 75358c021ebd4641ac3de3556179faab P a4642f7baed54c09994b275f9837d000 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4642f7baed54c09994b275f9837d000 (127.28.207.67). New cstate: current_term: 1 leader_uuid: "a4642f7baed54c09994b275f9837d000" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4642f7baed54c09994b275f9837d000" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 } health_report { overall_health: UNKNOWN } } }
I20260929 18:32:06.111361 31119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:06.107421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41806 (local address 127.28.207.67:37409)
0929 18:32:06.107602 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:06.107607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:06.107626 (+ 19us) server_negotiation.cc:407] Connection header received
0929 18:32:06.107713 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:06.107721 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:06.107784 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:06.107887 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:32:06.108181 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.108740 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.109694 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.109957 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.110072 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:06.110436 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:32:06.110930 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:32:06.111049 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:06.111155 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 18:32:06.111205 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 18:32:06.118369 30912 raft_consensus.cc:1261] T 75358c021ebd4641ac3de3556179faab P 3c52aae6cbe94546b373509a42d6ca54 [term 1 FOLLOWER]: Refusing update from remote peer a4642f7baed54c09994b275f9837d000: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:32:06.118645 30772 raft_consensus.cc:1261] T 75358c021ebd4641ac3de3556179faab P 28f15042e6c541e4b27b316554dbddfd [term 1 FOLLOWER]: Refusing update from remote peer a4642f7baed54c09994b275f9837d000: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:32:06.119150 31129 consensus_queue.cc:1035] T 75358c021ebd4641ac3de3556179faab P a4642f7baed54c09994b275f9837d000 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:32:06.119374 31135 consensus_queue.cc:1035] T 75358c021ebd4641ac3de3556179faab P a4642f7baed54c09994b275f9837d000 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:32:06.129006 31138 mvcc.cc:205] Tried to move back new op lower bound from 7334734750174298112 to 7334734749999144960. Current Snapshot: MvccSnapshot[applied={T|T < 7334734750174298112}]
I20260929 18:32:06.129774 31139 mvcc.cc:205] Tried to move back new op lower bound from 7334734750174298112 to 7334734749999144960. Current Snapshot: MvccSnapshot[applied={T|T < 7334734750174298112}]
I20260929 18:32:06.132361 31140 mvcc.cc:205] Tried to move back new op lower bound from 7334734750174298112 to 7334734749999144960. Current Snapshot: MvccSnapshot[applied={T|T < 7334734750174298112}]
I20260929 18:32:06.147027 30612 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:32:06.147218 30612 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:32:06.150182 30612 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a031dfb803c416a988640504a679e57: Sending DeleteTablet for 3 replicas of tablet 75358c021ebd4641ac3de3556179faab
I20260929 18:32:06.151580 30747 tablet_service.cc:1514] Processing DeleteTablet for tablet 75358c021ebd4641ac3de3556179faab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:06 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:50434
I20260929 18:32:06.151562 30893 tablet_service.cc:1514] Processing DeleteTablet for tablet 75358c021ebd4641ac3de3556179faab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:06 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:34022
I20260929 18:32:06.152259 31148 tablet_replica.cc:330] T 75358c021ebd4641ac3de3556179faab P 28f15042e6c541e4b27b316554dbddfd: stopping tablet replica
I20260929 18:32:06.152225 30612 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 18:32:06.152624 31148 raft_consensus.cc:2229] T 75358c021ebd4641ac3de3556179faab P 28f15042e6c541e4b27b316554dbddfd [term 1 FOLLOWER]: Raft consensus shutting down.
W20260929 18:32:06.152714 30612 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:32:06.152755 31033 tablet_service.cc:1514] Processing DeleteTablet for tablet 75358c021ebd4641ac3de3556179faab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:06 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:41792
I20260929 18:32:06.153147 31148 raft_consensus.cc:2258] T 75358c021ebd4641ac3de3556179faab P 28f15042e6c541e4b27b316554dbddfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:06.155148 31148 ts_tablet_manager.cc:1905] T 75358c021ebd4641ac3de3556179faab P 28f15042e6c541e4b27b316554dbddfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:32:06.157876 31149 tablet_replica.cc:330] T 75358c021ebd4641ac3de3556179faab P 3c52aae6cbe94546b373509a42d6ca54: stopping tablet replica
I20260929 18:32:06.158465 31149 raft_consensus.cc:2229] T 75358c021ebd4641ac3de3556179faab P 3c52aae6cbe94546b373509a42d6ca54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:32:06.159097 31148 ts_tablet_manager.cc:1918] T 75358c021ebd4641ac3de3556179faab P 28f15042e6c541e4b27b316554dbddfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:32:06.159235 31148 log.cc:1196] T 75358c021ebd4641ac3de3556179faab P 28f15042e6c541e4b27b316554dbddfd: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-0/wal/wals/75358c021ebd4641ac3de3556179faab
I20260929 18:32:06.159581 31148 ts_tablet_manager.cc:1939] T 75358c021ebd4641ac3de3556179faab P 28f15042e6c541e4b27b316554dbddfd: Deleting consensus metadata
I20260929 18:32:06.160339 30893 tablet_service.cc:1467] Processing CreateTablet for tablet f2af830f0afc4386a217d3bdd0a3abf0 (DEFAULT_TABLE table=test-table [id=2f72ccf2e9a243f3937693a663bf04d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:06.160586 31150 tablet_replica.cc:330] T 75358c021ebd4641ac3de3556179faab P a4642f7baed54c09994b275f9837d000: stopping tablet replica
I20260929 18:32:06.160660 30893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2af830f0afc4386a217d3bdd0a3abf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:06.161171 31150 raft_consensus.cc:2229] T 75358c021ebd4641ac3de3556179faab P a4642f7baed54c09994b275f9837d000 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:32:06.161966 31150 raft_consensus.cc:2258] T 75358c021ebd4641ac3de3556179faab P a4642f7baed54c09994b275f9837d000 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:06.162396 31149 raft_consensus.cc:2258] T 75358c021ebd4641ac3de3556179faab P 3c52aae6cbe94546b373509a42d6ca54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:06.163161 31033 tablet_service.cc:1467] Processing CreateTablet for tablet f2af830f0afc4386a217d3bdd0a3abf0 (DEFAULT_TABLE table=test-table [id=2f72ccf2e9a243f3937693a663bf04d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:06.163592 31033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2af830f0afc4386a217d3bdd0a3abf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:06.164844 31149 ts_tablet_manager.cc:1905] T 75358c021ebd4641ac3de3556179faab P 3c52aae6cbe94546b373509a42d6ca54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:32:06.166560 30602 catalog_manager.cc:4841] TS 28f15042e6c541e4b27b316554dbddfd (127.28.207.65:44095): tablet 75358c021ebd4641ac3de3556179faab (table test-table [id=ed05f75b28364f3d9ed9c6d064d9d07e]) successfully deleted
I20260929 18:32:06.167173 31125 tablet_bootstrap.cc:492] T f2af830f0afc4386a217d3bdd0a3abf0 P a4642f7baed54c09994b275f9837d000: Bootstrap starting.
I20260929 18:32:06.167678 31124 tablet_bootstrap.cc:492] T f2af830f0afc4386a217d3bdd0a3abf0 P 3c52aae6cbe94546b373509a42d6ca54: Bootstrap starting.
I20260929 18:32:06.168380 30747 tablet_service.cc:1467] Processing CreateTablet for tablet f2af830f0afc4386a217d3bdd0a3abf0 (DEFAULT_TABLE table=test-table [id=2f72ccf2e9a243f3937693a663bf04d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:06.168514 31125 tablet_bootstrap.cc:654] T f2af830f0afc4386a217d3bdd0a3abf0 P a4642f7baed54c09994b275f9837d000: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:06.168756 30747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2af830f0afc4386a217d3bdd0a3abf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:06.169320 31124 tablet_bootstrap.cc:654] T f2af830f0afc4386a217d3bdd0a3abf0 P 3c52aae6cbe94546b373509a42d6ca54: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:06.170459 31125 tablet_bootstrap.cc:492] T f2af830f0afc4386a217d3bdd0a3abf0 P a4642f7baed54c09994b275f9837d000: No bootstrap required, opened a new log
I20260929 18:32:06.170569 31125 ts_tablet_manager.cc:1397] T f2af830f0afc4386a217d3bdd0a3abf0 P a4642f7baed54c09994b275f9837d000: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:32:06.171183 31125 raft_consensus.cc:348] T f2af830f0afc4386a217d3bdd0a3abf0 P a4642f7baed54c09994b275f9837d000 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 } } peers { permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 } } peers { permanent_uuid: "a4642f7baed54c09994b275f9837d000" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37409 } }
I20260929 18:32:06.171329 31125 raft_consensus.cc:374] T f2af830f0afc4386a217d3bdd0a3abf0 P a4642f7baed54c09994b275f9837d000 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:06.171372 31125 raft_consensus.cc:729] T f2af830f0afc4386a217d3bdd0a3abf0 P a4642f7baed54c09994b275f9837d000 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4642f7baed54c09994b275f9837d000, State: Initialized, Role: FOLLOWER
I20260929 18:32:06.171530 31125 consensus_queue.cc:260] T f2af830f0afc4386a217d3bdd0a3abf0 P a4642f7baed54c09994b275f9837d000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 } } peers { permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 } } peers { permanent_uuid: "a4642f7baed54c09994b275f9837d000" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37409 } }
I20260929 18:32:06.171936 31125 ts_tablet_manager.cc:1428] T f2af830f0afc4386a217d3bdd0a3abf0 P a4642f7baed54c09994b275f9837d000: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 18:32:06.171988 31150 ts_tablet_manager.cc:1905] T 75358c021ebd4641ac3de3556179faab P a4642f7baed54c09994b275f9837d000: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:32:06.173130 31124 tablet_bootstrap.cc:492] T f2af830f0afc4386a217d3bdd0a3abf0 P 3c52aae6cbe94546b373509a42d6ca54: No bootstrap required, opened a new log
I20260929 18:32:06.173267 31124 ts_tablet_manager.cc:1397] T f2af830f0afc4386a217d3bdd0a3abf0 P 3c52aae6cbe94546b373509a42d6ca54: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260929 18:32:06.173939 31124 raft_consensus.cc:348] T f2af830f0afc4386a217d3bdd0a3abf0 P 3c52aae6cbe94546b373509a42d6ca54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 } } peers { permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 } } peers { permanent_uuid: "a4642f7baed54c09994b275f9837d000" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37409 } }
I20260929 18:32:06.174093 31124 raft_consensus.cc:374] T f2af830f0afc4386a217d3bdd0a3abf0 P 3c52aae6cbe94546b373509a42d6ca54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:06.174139 31124 raft_consensus.cc:729] T f2af830f0afc4386a217d3bdd0a3abf0 P 3c52aae6cbe94546b373509a42d6ca54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c52aae6cbe94546b373509a42d6ca54, State: Initialized, Role: FOLLOWER
I20260929 18:32:06.174492 31124 consensus_queue.cc:260] T f2af830f0afc4386a217d3bdd0a3abf0 P 3c52aae6cbe94546b373509a42d6ca54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 } } peers { permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 } } peers { permanent_uuid: "a4642f7baed54c09994b275f9837d000" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37409 } }
I20260929 18:32:06.176949 31124 ts_tablet_manager.cc:1428] T f2af830f0afc4386a217d3bdd0a3abf0 P 3c52aae6cbe94546b373509a42d6ca54: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260929 18:32:06.177142 31149 ts_tablet_manager.cc:1918] T 75358c021ebd4641ac3de3556179faab P 3c52aae6cbe94546b373509a42d6ca54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:32:06.177280 31149 log.cc:1196] T 75358c021ebd4641ac3de3556179faab P 3c52aae6cbe94546b373509a42d6ca54: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-1/wal/wals/75358c021ebd4641ac3de3556179faab
I20260929 18:32:06.177711 31149 ts_tablet_manager.cc:1939] T 75358c021ebd4641ac3de3556179faab P 3c52aae6cbe94546b373509a42d6ca54: Deleting consensus metadata
I20260929 18:32:06.178308 31122 tablet_bootstrap.cc:492] T f2af830f0afc4386a217d3bdd0a3abf0 P 28f15042e6c541e4b27b316554dbddfd: Bootstrap starting.
I20260929 18:32:06.179664 30601 catalog_manager.cc:4841] TS 3c52aae6cbe94546b373509a42d6ca54 (127.28.207.66:38011): tablet 75358c021ebd4641ac3de3556179faab (table test-table [id=ed05f75b28364f3d9ed9c6d064d9d07e]) successfully deleted
I20260929 18:32:06.179872 31150 ts_tablet_manager.cc:1918] T 75358c021ebd4641ac3de3556179faab P a4642f7baed54c09994b275f9837d000: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:32:06.179936 31122 tablet_bootstrap.cc:654] T f2af830f0afc4386a217d3bdd0a3abf0 P 28f15042e6c541e4b27b316554dbddfd: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:06.180007 31150 log.cc:1196] T 75358c021ebd4641ac3de3556179faab P a4642f7baed54c09994b275f9837d000: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-2/wal/wals/75358c021ebd4641ac3de3556179faab
I20260929 18:32:06.180342 31150 ts_tablet_manager.cc:1939] T 75358c021ebd4641ac3de3556179faab P a4642f7baed54c09994b275f9837d000: Deleting consensus metadata
I20260929 18:32:06.181327 31122 tablet_bootstrap.cc:492] T f2af830f0afc4386a217d3bdd0a3abf0 P 28f15042e6c541e4b27b316554dbddfd: No bootstrap required, opened a new log
I20260929 18:32:06.181455 31122 ts_tablet_manager.cc:1397] T f2af830f0afc4386a217d3bdd0a3abf0 P 28f15042e6c541e4b27b316554dbddfd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:32:06.181964 31122 raft_consensus.cc:348] T f2af830f0afc4386a217d3bdd0a3abf0 P 28f15042e6c541e4b27b316554dbddfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 } } peers { permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 } } peers { permanent_uuid: "a4642f7baed54c09994b275f9837d000" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37409 } }
I20260929 18:32:06.181952 30601 catalog_manager.cc:4841] TS a4642f7baed54c09994b275f9837d000 (127.28.207.67:37409): tablet 75358c021ebd4641ac3de3556179faab (table test-table [id=ed05f75b28364f3d9ed9c6d064d9d07e]) successfully deleted
I20260929 18:32:06.182097 31122 raft_consensus.cc:374] T f2af830f0afc4386a217d3bdd0a3abf0 P 28f15042e6c541e4b27b316554dbddfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:06.182142 31122 raft_consensus.cc:729] T f2af830f0afc4386a217d3bdd0a3abf0 P 28f15042e6c541e4b27b316554dbddfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28f15042e6c541e4b27b316554dbddfd, State: Initialized, Role: FOLLOWER
I20260929 18:32:06.182286 31122 consensus_queue.cc:260] T f2af830f0afc4386a217d3bdd0a3abf0 P 28f15042e6c541e4b27b316554dbddfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 } } peers { permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 } } peers { permanent_uuid: "a4642f7baed54c09994b275f9837d000" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37409 } }
I20260929 18:32:06.182583 31122 ts_tablet_manager.cc:1428] T f2af830f0afc4386a217d3bdd0a3abf0 P 28f15042e6c541e4b27b316554dbddfd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:32:06.316198 31143 raft_consensus.cc:482] T f2af830f0afc4386a217d3bdd0a3abf0 P a4642f7baed54c09994b275f9837d000 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:32:06.316372 31143 raft_consensus.cc:504] T f2af830f0afc4386a217d3bdd0a3abf0 P a4642f7baed54c09994b275f9837d000 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 } } peers { permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 } } peers { permanent_uuid: "a4642f7baed54c09994b275f9837d000" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37409 } }
I20260929 18:32:06.316973 31143 leader_election.cc:288] T f2af830f0afc4386a217d3bdd0a3abf0 P a4642f7baed54c09994b275f9837d000 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28f15042e6c541e4b27b316554dbddfd (127.28.207.65:44095), 3c52aae6cbe94546b373509a42d6ca54 (127.28.207.66:38011)
I20260929 18:32:06.317617 30912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2af830f0afc4386a217d3bdd0a3abf0" candidate_uuid: "a4642f7baed54c09994b275f9837d000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c52aae6cbe94546b373509a42d6ca54" is_pre_election: true
I20260929 18:32:06.317647 30772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2af830f0afc4386a217d3bdd0a3abf0" candidate_uuid: "a4642f7baed54c09994b275f9837d000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28f15042e6c541e4b27b316554dbddfd" is_pre_election: true
I20260929 18:32:06.317802 30912 raft_consensus.cc:2454] T f2af830f0afc4386a217d3bdd0a3abf0 P 3c52aae6cbe94546b373509a42d6ca54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4642f7baed54c09994b275f9837d000 in term 0.
I20260929 18:32:06.317917 30772 raft_consensus.cc:2454] T f2af830f0afc4386a217d3bdd0a3abf0 P 28f15042e6c541e4b27b316554dbddfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4642f7baed54c09994b275f9837d000 in term 0.
I20260929 18:32:06.318224 30985 leader_election.cc:302] T f2af830f0afc4386a217d3bdd0a3abf0 P a4642f7baed54c09994b275f9837d000 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c52aae6cbe94546b373509a42d6ca54, a4642f7baed54c09994b275f9837d000; no voters:
I20260929 18:32:06.318449 31143 raft_consensus.cc:2781] T f2af830f0afc4386a217d3bdd0a3abf0 P a4642f7baed54c09994b275f9837d000 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:32:06.318538 31143 raft_consensus.cc:482] T f2af830f0afc4386a217d3bdd0a3abf0 P a4642f7baed54c09994b275f9837d000 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:32:06.318596 31143 raft_consensus.cc:3037] T f2af830f0afc4386a217d3bdd0a3abf0 P a4642f7baed54c09994b275f9837d000 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:06.319968 31143 raft_consensus.cc:504] T f2af830f0afc4386a217d3bdd0a3abf0 P a4642f7baed54c09994b275f9837d000 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 } } peers { permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 } } peers { permanent_uuid: "a4642f7baed54c09994b275f9837d000" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37409 } }
I20260929 18:32:06.320415 31143 leader_election.cc:288] T f2af830f0afc4386a217d3bdd0a3abf0 P a4642f7baed54c09994b275f9837d000 [CANDIDATE]: Term 1 election: Requested vote from peers 28f15042e6c541e4b27b316554dbddfd (127.28.207.65:44095), 3c52aae6cbe94546b373509a42d6ca54 (127.28.207.66:38011)
I20260929 18:32:06.320832 30912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2af830f0afc4386a217d3bdd0a3abf0" candidate_uuid: "a4642f7baed54c09994b275f9837d000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c52aae6cbe94546b373509a42d6ca54"
I20260929 18:32:06.320890 30772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2af830f0afc4386a217d3bdd0a3abf0" candidate_uuid: "a4642f7baed54c09994b275f9837d000" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28f15042e6c541e4b27b316554dbddfd"
I20260929 18:32:06.320983 30912 raft_consensus.cc:3037] T f2af830f0afc4386a217d3bdd0a3abf0 P 3c52aae6cbe94546b373509a42d6ca54 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:06.321084 30772 raft_consensus.cc:3037] T f2af830f0afc4386a217d3bdd0a3abf0 P 28f15042e6c541e4b27b316554dbddfd [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:06.322222 30912 raft_consensus.cc:2454] T f2af830f0afc4386a217d3bdd0a3abf0 P 3c52aae6cbe94546b373509a42d6ca54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4642f7baed54c09994b275f9837d000 in term 1.
I20260929 18:32:06.322583 30985 leader_election.cc:302] T f2af830f0afc4386a217d3bdd0a3abf0 P a4642f7baed54c09994b275f9837d000 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c52aae6cbe94546b373509a42d6ca54, a4642f7baed54c09994b275f9837d000; no voters:
I20260929 18:32:06.322671 30772 raft_consensus.cc:2454] T f2af830f0afc4386a217d3bdd0a3abf0 P 28f15042e6c541e4b27b316554dbddfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4642f7baed54c09994b275f9837d000 in term 1.
I20260929 18:32:06.322809 31143 raft_consensus.cc:2781] T f2af830f0afc4386a217d3bdd0a3abf0 P a4642f7baed54c09994b275f9837d000 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:32:06.322988 31143 raft_consensus.cc:686] T f2af830f0afc4386a217d3bdd0a3abf0 P a4642f7baed54c09994b275f9837d000 [term 1 LEADER]: Becoming Leader. State: Replica: a4642f7baed54c09994b275f9837d000, State: Running, Role: LEADER
I20260929 18:32:06.323179 31143 consensus_queue.cc:237] T f2af830f0afc4386a217d3bdd0a3abf0 P a4642f7baed54c09994b275f9837d000 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 } } peers { permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 } } peers { permanent_uuid: "a4642f7baed54c09994b275f9837d000" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37409 } }
I20260929 18:32:06.325137 30616 catalog_manager.cc:5495] T f2af830f0afc4386a217d3bdd0a3abf0 P a4642f7baed54c09994b275f9837d000 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4642f7baed54c09994b275f9837d000 (127.28.207.67). New cstate: current_term: 1 leader_uuid: "a4642f7baed54c09994b275f9837d000" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4642f7baed54c09994b275f9837d000" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37409 } health_report { overall_health: HEALTHY } } }
Sep 29 18:32:06 dist-test-slave-qcpv krb5kdc[30571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706724, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:32:06.405097 31160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:06.393579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36751 (local address 127.0.0.1:41736)
0929 18:32:06.393999 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:06.394039 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:06.394183 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:06.394716 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:06.394719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:06.395309 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:06.395605 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:06.395619 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.396848 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.396851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:06.397494 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:06.397502 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.398050 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.398053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:06.398127 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:06.398958 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:06.398973 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:06.401207 (+ 2234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:06.403360 (+ 2153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:06.403374 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:06.403382 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:06.403629 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:06.403920 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:06.403925 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:06.403927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:06.404003 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:06.404409 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:06.404419 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:06.404523 (+ 104us) client_negotiation.cc:769] Sending connection context
0929 18:32:06.404641 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 18:32:06.404870 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":149,"threads_started":1}
I20260929 18:32:06.405097 31161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:06.393750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41736 (local address 127.28.207.126:36751)
0929 18:32:06.394134 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:06.394141 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:06.394162 (+ 21us) server_negotiation.cc:407] Connection header received
0929 18:32:06.394363 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:06.394366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:06.394446 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:06.394589 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:06.396061 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.396670 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.397634 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.397888 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.397979 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:06.401380 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:06.401383 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:06.401385 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:06.401406 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:06.403245 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:06.403741 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:06.403747 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:06.403749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:06.403827 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:06.404095 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:06.404100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:06.404102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:06.404292 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:06.404398 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:06.404699 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 18:32:06.404923 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":112,"server-negotiator.queue_time_us":302,"thread_start_us":147,"threads_started":1}
I20260929 18:32:06.410683 30616 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:41736:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 18:32:06.411722 30616 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:32:06.418309 30893 tablet_service.cc:1467] Processing CreateTablet for tablet d6baf2e236f345ea92ef7d45691fc676 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e37cb69a70454e938ac5c84dbf142133]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:32:06.418309 30747 tablet_service.cc:1467] Processing CreateTablet for tablet d6baf2e236f345ea92ef7d45691fc676 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e37cb69a70454e938ac5c84dbf142133]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:32:06.418757 30747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6baf2e236f345ea92ef7d45691fc676. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:06.418757 30893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6baf2e236f345ea92ef7d45691fc676. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:06.418638 31033 tablet_service.cc:1467] Processing CreateTablet for tablet d6baf2e236f345ea92ef7d45691fc676 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e37cb69a70454e938ac5c84dbf142133]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:32:06.419173 31033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6baf2e236f345ea92ef7d45691fc676. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:06.422009 31124 tablet_bootstrap.cc:492] T d6baf2e236f345ea92ef7d45691fc676 P 3c52aae6cbe94546b373509a42d6ca54: Bootstrap starting.
I20260929 18:32:06.422925 31125 tablet_bootstrap.cc:492] T d6baf2e236f345ea92ef7d45691fc676 P a4642f7baed54c09994b275f9837d000: Bootstrap starting.
I20260929 18:32:06.423345 31124 tablet_bootstrap.cc:654] T d6baf2e236f345ea92ef7d45691fc676 P 3c52aae6cbe94546b373509a42d6ca54: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:06.423938 31125 tablet_bootstrap.cc:654] T d6baf2e236f345ea92ef7d45691fc676 P a4642f7baed54c09994b275f9837d000: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:06.424372 31122 tablet_bootstrap.cc:492] T d6baf2e236f345ea92ef7d45691fc676 P 28f15042e6c541e4b27b316554dbddfd: Bootstrap starting.
I20260929 18:32:06.425004 31124 tablet_bootstrap.cc:492] T d6baf2e236f345ea92ef7d45691fc676 P 3c52aae6cbe94546b373509a42d6ca54: No bootstrap required, opened a new log
I20260929 18:32:06.425076 31124 ts_tablet_manager.cc:1397] T d6baf2e236f345ea92ef7d45691fc676 P 3c52aae6cbe94546b373509a42d6ca54: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:32:06.425300 31122 tablet_bootstrap.cc:654] T d6baf2e236f345ea92ef7d45691fc676 P 28f15042e6c541e4b27b316554dbddfd: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:06.425465 31124 raft_consensus.cc:348] T d6baf2e236f345ea92ef7d45691fc676 P 3c52aae6cbe94546b373509a42d6ca54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 } } peers { permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 } } peers { permanent_uuid: "a4642f7baed54c09994b275f9837d000" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37409 } }
I20260929 18:32:06.425554 31124 raft_consensus.cc:374] T d6baf2e236f345ea92ef7d45691fc676 P 3c52aae6cbe94546b373509a42d6ca54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:06.425575 31124 raft_consensus.cc:729] T d6baf2e236f345ea92ef7d45691fc676 P 3c52aae6cbe94546b373509a42d6ca54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c52aae6cbe94546b373509a42d6ca54, State: Initialized, Role: FOLLOWER
I20260929 18:32:06.425753 31124 consensus_queue.cc:260] T d6baf2e236f345ea92ef7d45691fc676 P 3c52aae6cbe94546b373509a42d6ca54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 } } peers { permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 } } peers { permanent_uuid: "a4642f7baed54c09994b275f9837d000" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37409 } }
I20260929 18:32:06.426044 31124 ts_tablet_manager.cc:1428] T d6baf2e236f345ea92ef7d45691fc676 P 3c52aae6cbe94546b373509a42d6ca54: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 18:32:06.425997 31130 tablet_replica.cc:439] T d6baf2e236f345ea92ef7d45691fc676 P 3c52aae6cbe94546b373509a42d6ca54: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 } } peers { permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 } } peers { permanent_uuid: "a4642f7baed54c09994b275f9837d000" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37409 } } }
I20260929 18:32:06.426133 31130 tablet_replica.cc:442] T d6baf2e236f345ea92ef7d45691fc676 P 3c52aae6cbe94546b373509a42d6ca54: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:32:06.426687 31122 tablet_bootstrap.cc:492] T d6baf2e236f345ea92ef7d45691fc676 P 28f15042e6c541e4b27b316554dbddfd: No bootstrap required, opened a new log
I20260929 18:32:06.426815 31122 ts_tablet_manager.cc:1397] T d6baf2e236f345ea92ef7d45691fc676 P 28f15042e6c541e4b27b316554dbddfd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:32:06.427336 31122 raft_consensus.cc:348] T d6baf2e236f345ea92ef7d45691fc676 P 28f15042e6c541e4b27b316554dbddfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 } } peers { permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 } } peers { permanent_uuid: "a4642f7baed54c09994b275f9837d000" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37409 } }
I20260929 18:32:06.427517 31122 raft_consensus.cc:374] T d6baf2e236f345ea92ef7d45691fc676 P 28f15042e6c541e4b27b316554dbddfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:06.427552 31122 raft_consensus.cc:729] T d6baf2e236f345ea92ef7d45691fc676 P 28f15042e6c541e4b27b316554dbddfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28f15042e6c541e4b27b316554dbddfd, State: Initialized, Role: FOLLOWER
I20260929 18:32:06.427736 31122 consensus_queue.cc:260] T d6baf2e236f345ea92ef7d45691fc676 P 28f15042e6c541e4b27b316554dbddfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 } } peers { permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 } } peers { permanent_uuid: "a4642f7baed54c09994b275f9837d000" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37409 } }
I20260929 18:32:06.428025 31127 tablet_replica.cc:439] T d6baf2e236f345ea92ef7d45691fc676 P 28f15042e6c541e4b27b316554dbddfd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 } } peers { permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 } } peers { permanent_uuid: "a4642f7baed54c09994b275f9837d000" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37409 } } }
I20260929 18:32:06.428174 31127 tablet_replica.cc:442] T d6baf2e236f345ea92ef7d45691fc676 P 28f15042e6c541e4b27b316554dbddfd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:32:06.428507 31122 ts_tablet_manager.cc:1428] T d6baf2e236f345ea92ef7d45691fc676 P 28f15042e6c541e4b27b316554dbddfd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:32:06.429682 31125 tablet_bootstrap.cc:492] T d6baf2e236f345ea92ef7d45691fc676 P a4642f7baed54c09994b275f9837d000: No bootstrap required, opened a new log
I20260929 18:32:06.429819 31125 ts_tablet_manager.cc:1397] T d6baf2e236f345ea92ef7d45691fc676 P a4642f7baed54c09994b275f9837d000: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260929 18:32:06.430233 31125 raft_consensus.cc:348] T d6baf2e236f345ea92ef7d45691fc676 P a4642f7baed54c09994b275f9837d000 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 } } peers { permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 } } peers { permanent_uuid: "a4642f7baed54c09994b275f9837d000" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37409 } }
I20260929 18:32:06.430372 31125 raft_consensus.cc:374] T d6baf2e236f345ea92ef7d45691fc676 P a4642f7baed54c09994b275f9837d000 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:06.430426 31125 raft_consensus.cc:729] T d6baf2e236f345ea92ef7d45691fc676 P a4642f7baed54c09994b275f9837d000 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4642f7baed54c09994b275f9837d000, State: Initialized, Role: FOLLOWER
I20260929 18:32:06.430562 31125 consensus_queue.cc:260] T d6baf2e236f345ea92ef7d45691fc676 P a4642f7baed54c09994b275f9837d000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 } } peers { permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 } } peers { permanent_uuid: "a4642f7baed54c09994b275f9837d000" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37409 } }
I20260929 18:32:06.430812 31143 tablet_replica.cc:439] T d6baf2e236f345ea92ef7d45691fc676 P a4642f7baed54c09994b275f9837d000: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 } } peers { permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 } } peers { permanent_uuid: "a4642f7baed54c09994b275f9837d000" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37409 } } }
I20260929 18:32:06.431000 31143 tablet_replica.cc:442] T d6baf2e236f345ea92ef7d45691fc676 P a4642f7baed54c09994b275f9837d000: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:32:06.431310 31125 ts_tablet_manager.cc:1428] T d6baf2e236f345ea92ef7d45691fc676 P a4642f7baed54c09994b275f9837d000: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:32:06.464591 31130 raft_consensus.cc:482] T d6baf2e236f345ea92ef7d45691fc676 P 3c52aae6cbe94546b373509a42d6ca54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:32:06.464754 31130 raft_consensus.cc:504] T d6baf2e236f345ea92ef7d45691fc676 P 3c52aae6cbe94546b373509a42d6ca54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 } } peers { permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 } } peers { permanent_uuid: "a4642f7baed54c09994b275f9837d000" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37409 } }
I20260929 18:32:06.466002 31130 leader_election.cc:288] T d6baf2e236f345ea92ef7d45691fc676 P 3c52aae6cbe94546b373509a42d6ca54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28f15042e6c541e4b27b316554dbddfd (127.28.207.65:44095), a4642f7baed54c09994b275f9837d000 (127.28.207.67:37409)
I20260929 18:32:06.471431 31119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:06.466540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:44195 (local address 127.28.207.67:37409)
0929 18:32:06.466669 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:06.466674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:06.466989 (+ 315us) server_negotiation.cc:407] Connection header received
0929 18:32:06.467350 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:06.467354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:06.467424 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:06.467533 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:06.467988 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.468641 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.470262 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.470674 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.470825 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:06.470937 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:06.471139 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 18:32:06.471248 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:32:06.471534 31118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:06.466440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:52833 (local address 127.28.207.65:44095)
0929 18:32:06.466633 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:06.466639 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:06.466885 (+ 246us) server_negotiation.cc:407] Connection header received
0929 18:32:06.467122 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:06.467127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:06.467193 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:06.467257 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:06.467752 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.468352 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.470223 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.470832 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.471010 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:06.471075 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:06.471273 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 18:32:06.471387 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 18:32:06.471932 31166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:06.466468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:37409 (local address 127.28.207.66:44195)
0929 18:32:06.466847 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:06.466866 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:06.467067 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:06.467554 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:06.467557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:06.467576 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:06.467787 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:06.467794 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.468821 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.468824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:06.470062 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:06.470078 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.470820 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.470824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:06.470977 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:06.471007 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 18:32:06.471631 (+ 624us) client_negotiation.cc:240] Negotiation successful
0929 18:32:06.471750 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":22,"thread_start_us":128,"threads_started":1}
I20260929 18:32:06.472296 31165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:06.466216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:44095 (local address 127.28.207.66:52833)
0929 18:32:06.466713 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:06.466732 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:06.466978 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:06.467345 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:06.467349 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:06.467371 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:06.467572 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:06.467580 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.468489 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.468493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:06.470098 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:06.470105 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.471007 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.471010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:06.471123 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:06.471141 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 18:32:06.471813 (+ 672us) client_negotiation.cc:240] Negotiation successful
0929 18:32:06.471886 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":91,"thread_start_us":144,"threads_started":1}
I20260929 18:32:06.472532 31053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6baf2e236f345ea92ef7d45691fc676" candidate_uuid: "3c52aae6cbe94546b373509a42d6ca54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4642f7baed54c09994b275f9837d000" is_pre_election: true
I20260929 18:32:06.472728 31053 raft_consensus.cc:2454] T d6baf2e236f345ea92ef7d45691fc676 P a4642f7baed54c09994b275f9837d000 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c52aae6cbe94546b373509a42d6ca54 in term 0.
I20260929 18:32:06.472889 30772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6baf2e236f345ea92ef7d45691fc676" candidate_uuid: "3c52aae6cbe94546b373509a42d6ca54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28f15042e6c541e4b27b316554dbddfd" is_pre_election: true
I20260929 18:32:06.473078 30772 raft_consensus.cc:2454] T d6baf2e236f345ea92ef7d45691fc676 P 28f15042e6c541e4b27b316554dbddfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c52aae6cbe94546b373509a42d6ca54 in term 0.
I20260929 18:32:06.473937 30845 leader_election.cc:302] T d6baf2e236f345ea92ef7d45691fc676 P 3c52aae6cbe94546b373509a42d6ca54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c52aae6cbe94546b373509a42d6ca54, a4642f7baed54c09994b275f9837d000; no voters:
I20260929 18:32:06.474390 31130 raft_consensus.cc:2781] T d6baf2e236f345ea92ef7d45691fc676 P 3c52aae6cbe94546b373509a42d6ca54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:32:06.474500 31130 raft_consensus.cc:482] T d6baf2e236f345ea92ef7d45691fc676 P 3c52aae6cbe94546b373509a42d6ca54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:32:06.474550 31130 raft_consensus.cc:3037] T d6baf2e236f345ea92ef7d45691fc676 P 3c52aae6cbe94546b373509a42d6ca54 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:06.475769 31130 raft_consensus.cc:504] T d6baf2e236f345ea92ef7d45691fc676 P 3c52aae6cbe94546b373509a42d6ca54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 } } peers { permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 } } peers { permanent_uuid: "a4642f7baed54c09994b275f9837d000" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37409 } }
I20260929 18:32:06.476357 31130 leader_election.cc:288] T d6baf2e236f345ea92ef7d45691fc676 P 3c52aae6cbe94546b373509a42d6ca54 [CANDIDATE]: Term 1 election: Requested vote from peers 28f15042e6c541e4b27b316554dbddfd (127.28.207.65:44095), a4642f7baed54c09994b275f9837d000 (127.28.207.67:37409)
I20260929 18:32:06.476753 30772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6baf2e236f345ea92ef7d45691fc676" candidate_uuid: "3c52aae6cbe94546b373509a42d6ca54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28f15042e6c541e4b27b316554dbddfd"
I20260929 18:32:06.476904 30772 raft_consensus.cc:3037] T d6baf2e236f345ea92ef7d45691fc676 P 28f15042e6c541e4b27b316554dbddfd [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:06.477068 31053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6baf2e236f345ea92ef7d45691fc676" candidate_uuid: "3c52aae6cbe94546b373509a42d6ca54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4642f7baed54c09994b275f9837d000"
I20260929 18:32:06.477210 31053 raft_consensus.cc:3037] T d6baf2e236f345ea92ef7d45691fc676 P a4642f7baed54c09994b275f9837d000 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:06.478011 30772 raft_consensus.cc:2454] T d6baf2e236f345ea92ef7d45691fc676 P 28f15042e6c541e4b27b316554dbddfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c52aae6cbe94546b373509a42d6ca54 in term 1.
I20260929 18:32:06.478479 30846 leader_election.cc:302] T d6baf2e236f345ea92ef7d45691fc676 P 3c52aae6cbe94546b373509a42d6ca54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28f15042e6c541e4b27b316554dbddfd, 3c52aae6cbe94546b373509a42d6ca54; no voters:
I20260929 18:32:06.478621 31053 raft_consensus.cc:2454] T d6baf2e236f345ea92ef7d45691fc676 P a4642f7baed54c09994b275f9837d000 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c52aae6cbe94546b373509a42d6ca54 in term 1.
I20260929 18:32:06.478754 31130 raft_consensus.cc:2781] T d6baf2e236f345ea92ef7d45691fc676 P 3c52aae6cbe94546b373509a42d6ca54 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:32:06.479058 31130 raft_consensus.cc:686] T d6baf2e236f345ea92ef7d45691fc676 P 3c52aae6cbe94546b373509a42d6ca54 [term 1 LEADER]: Becoming Leader. State: Replica: 3c52aae6cbe94546b373509a42d6ca54, State: Running, Role: LEADER
I20260929 18:32:06.479526 31130 consensus_queue.cc:237] T d6baf2e236f345ea92ef7d45691fc676 P 3c52aae6cbe94546b373509a42d6ca54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 } } peers { permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 } } peers { permanent_uuid: "a4642f7baed54c09994b275f9837d000" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37409 } }
I20260929 18:32:06.480681 31167 tablet_replica.cc:439] T d6baf2e236f345ea92ef7d45691fc676 P 3c52aae6cbe94546b373509a42d6ca54: TxnStatusTablet state changed. Reason: New leader 3c52aae6cbe94546b373509a42d6ca54. Latest consensus state: current_term: 1 leader_uuid: "3c52aae6cbe94546b373509a42d6ca54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 } } peers { permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 } } peers { permanent_uuid: "a4642f7baed54c09994b275f9837d000" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37409 } } }
I20260929 18:32:06.480793 31167 tablet_replica.cc:442] T d6baf2e236f345ea92ef7d45691fc676 P 3c52aae6cbe94546b373509a42d6ca54: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:32:06.482543 30616 catalog_manager.cc:5495] T d6baf2e236f345ea92ef7d45691fc676 P 3c52aae6cbe94546b373509a42d6ca54 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c52aae6cbe94546b373509a42d6ca54 (127.28.207.66). New cstate: current_term: 1 leader_uuid: "3c52aae6cbe94546b373509a42d6ca54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4642f7baed54c09994b275f9837d000" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37409 } health_report { overall_health: UNKNOWN } } }
I20260929 18:32:06.484606 31169 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 18:32:06.484818 31169 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 18:32:06.514168 31160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:06.509185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:38011 (local address 127.0.0.1:34036)
0929 18:32:06.509547 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:06.509572 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:06.509719 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:06.510152 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:06.510155 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:06.510163 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:32:06.510298 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:06.510303 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.511414 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.511417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:06.512119 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:06.512128 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.512783 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.512787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:06.512867 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:06.512929 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:32:06.513811 (+ 882us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:32:06.513828 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 18:32:06.513906 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 18:32:06.514010 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131}
I20260929 18:32:06.514176 31134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:06.509390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34036 (local address 127.28.207.66:38011)
0929 18:32:06.509670 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:06.509676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:06.509696 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:32:06.509894 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:06.509898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:06.509975 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:06.510101 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:32:06.510418 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.511239 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.512261 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.512618 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.512733 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:06.513104 (+ 371us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:32:06.513681 (+ 577us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:32:06.513798 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:06.513953 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 18:32:06.514015 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260929 18:32:06.518025 31053 raft_consensus.cc:1261] T d6baf2e236f345ea92ef7d45691fc676 P a4642f7baed54c09994b275f9837d000 [term 1 FOLLOWER]: Refusing update from remote peer 3c52aae6cbe94546b373509a42d6ca54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:32:06.518362 30772 raft_consensus.cc:1261] T d6baf2e236f345ea92ef7d45691fc676 P 28f15042e6c541e4b27b316554dbddfd [term 1 FOLLOWER]: Refusing update from remote peer 3c52aae6cbe94546b373509a42d6ca54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:32:06.518875 31167 consensus_queue.cc:1035] T d6baf2e236f345ea92ef7d45691fc676 P 3c52aae6cbe94546b373509a42d6ca54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4642f7baed54c09994b275f9837d000" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:32:06.519124 31130 consensus_queue.cc:1035] T d6baf2e236f345ea92ef7d45691fc676 P 3c52aae6cbe94546b373509a42d6ca54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:32:06.521006 31143 tablet_replica.cc:439] T d6baf2e236f345ea92ef7d45691fc676 P a4642f7baed54c09994b275f9837d000: TxnStatusTablet state changed. Reason: New leader 3c52aae6cbe94546b373509a42d6ca54. Latest consensus state: current_term: 1 leader_uuid: "3c52aae6cbe94546b373509a42d6ca54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 } } peers { permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 } } peers { permanent_uuid: "a4642f7baed54c09994b275f9837d000" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37409 } } }
I20260929 18:32:06.521129 31143 tablet_replica.cc:442] T d6baf2e236f345ea92ef7d45691fc676 P a4642f7baed54c09994b275f9837d000: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:32:06.521104 31127 tablet_replica.cc:439] T d6baf2e236f345ea92ef7d45691fc676 P 28f15042e6c541e4b27b316554dbddfd: TxnStatusTablet state changed. Reason: New leader 3c52aae6cbe94546b373509a42d6ca54. Latest consensus state: current_term: 1 leader_uuid: "3c52aae6cbe94546b373509a42d6ca54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 } } peers { permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 } } peers { permanent_uuid: "a4642f7baed54c09994b275f9837d000" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37409 } } }
I20260929 18:32:06.521240 31127 tablet_replica.cc:442] T d6baf2e236f345ea92ef7d45691fc676 P 28f15042e6c541e4b27b316554dbddfd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:32:06.524307 31130 tablet_replica.cc:439] T d6baf2e236f345ea92ef7d45691fc676 P 3c52aae6cbe94546b373509a42d6ca54: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3c52aae6cbe94546b373509a42d6ca54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 } } peers { permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 } } peers { permanent_uuid: "a4642f7baed54c09994b275f9837d000" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37409 } } }
I20260929 18:32:06.524470 31130 tablet_replica.cc:442] T d6baf2e236f345ea92ef7d45691fc676 P 3c52aae6cbe94546b373509a42d6ca54: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:32:06.525653 31167 tablet_replica.cc:439] T d6baf2e236f345ea92ef7d45691fc676 P 3c52aae6cbe94546b373509a42d6ca54: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3c52aae6cbe94546b373509a42d6ca54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 } } peers { permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 } } peers { permanent_uuid: "a4642f7baed54c09994b275f9837d000" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37409 } } }
I20260929 18:32:06.525790 31167 tablet_replica.cc:442] T d6baf2e236f345ea92ef7d45691fc676 P 3c52aae6cbe94546b373509a42d6ca54: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:32:06.526829 31143 tablet_replica.cc:439] T d6baf2e236f345ea92ef7d45691fc676 P a4642f7baed54c09994b275f9837d000: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3c52aae6cbe94546b373509a42d6ca54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 } } peers { permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 } } peers { permanent_uuid: "a4642f7baed54c09994b275f9837d000" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37409 } } }
I20260929 18:32:06.526981 31143 tablet_replica.cc:442] T d6baf2e236f345ea92ef7d45691fc676 P a4642f7baed54c09994b275f9837d000: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:32:06.528838 31127 tablet_replica.cc:439] T d6baf2e236f345ea92ef7d45691fc676 P 28f15042e6c541e4b27b316554dbddfd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3c52aae6cbe94546b373509a42d6ca54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 } } peers { permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 } } peers { permanent_uuid: "a4642f7baed54c09994b275f9837d000" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37409 } } }
I20260929 18:32:06.528998 31127 tablet_replica.cc:442] T d6baf2e236f345ea92ef7d45691fc676 P 28f15042e6c541e4b27b316554dbddfd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:32:06.543838 31134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:06.539002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:57045 (local address 127.28.207.66:38011)
0929 18:32:06.539198 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:06.539203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:06.539249 (+ 46us) server_negotiation.cc:407] Connection header received
0929 18:32:06.539739 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:06.539742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:06.539812 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:06.539915 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:32:06.540318 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.541081 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.542417 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.542669 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.542788 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:06.543203 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:32:06.543318 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:32:06.543491 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:06.543619 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 18:32:06.543685 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 18:32:06.543882 31176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:06.538743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:38011 (local address 127.28.207.67:57045)
0929 18:32:06.539151 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:06.539168 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:06.539324 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:06.540040 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:06.540043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:06.540055 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:32:06.540173 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:06.540180 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.541213 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.541216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:06.542253 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:06.542263 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.542823 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.542826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:06.542946 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:06.543016 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:32:06.543505 (+ 489us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:32:06.543512 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 18:32:06.543593 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 18:32:06.543699 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":140,"threads_started":1}
I20260929 18:32:06.553254 30772 raft_consensus.cc:1261] T f2af830f0afc4386a217d3bdd0a3abf0 P 28f15042e6c541e4b27b316554dbddfd [term 1 FOLLOWER]: Refusing update from remote peer a4642f7baed54c09994b275f9837d000: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:32:06.553313 30912 raft_consensus.cc:1261] T f2af830f0afc4386a217d3bdd0a3abf0 P 3c52aae6cbe94546b373509a42d6ca54 [term 1 FOLLOWER]: Refusing update from remote peer a4642f7baed54c09994b275f9837d000: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:32:06.553769 31143 consensus_queue.cc:1035] T f2af830f0afc4386a217d3bdd0a3abf0 P a4642f7baed54c09994b275f9837d000 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28f15042e6c541e4b27b316554dbddfd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 44095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:32:06.554023 31129 consensus_queue.cc:1035] T f2af830f0afc4386a217d3bdd0a3abf0 P a4642f7baed54c09994b275f9837d000 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c52aae6cbe94546b373509a42d6ca54" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:32:06.583266 31119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:06.577984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:44139 (local address 127.28.207.67:37409)
0929 18:32:06.578242 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:06.578248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:06.578781 (+ 533us) server_negotiation.cc:407] Connection header received
0929 18:32:06.579019 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:06.579022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:06.579076 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:06.579125 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:32:06.579572 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.580197 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.581848 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.582116 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.582226 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:06.582658 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:32:06.582748 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:32:06.582913 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:06.583053 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 18:32:06.583116 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260929 18:32:06.583281 31181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:06.577826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:37409 (local address 127.28.207.66:44139)
0929 18:32:06.578697 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:06.578715 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:06.578889 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:06.579264 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:06.579267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:06.579275 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:32:06.579407 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:06.579413 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.580345 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.580348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:06.581682 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:06.581696 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.582275 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.582279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:06.582381 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:06.582448 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:32:06.582921 (+ 473us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:32:06.582930 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 18:32:06.583007 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 18:32:06.583097 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"thread_start_us":230,"threads_started":1}
I20260929 18:32:06.615748 30614 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:32:06.615986 30614 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:32:06.619606 30614 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a031dfb803c416a988640504a679e57: Sending DeleteTablet for 3 replicas of tablet f2af830f0afc4386a217d3bdd0a3abf0
I20260929 18:32:06.620843 30747 tablet_service.cc:1514] Processing DeleteTablet for tablet f2af830f0afc4386a217d3bdd0a3abf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:06 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:50434
I20260929 18:32:06.621081 31148 tablet_replica.cc:330] T f2af830f0afc4386a217d3bdd0a3abf0 P 28f15042e6c541e4b27b316554dbddfd: stopping tablet replica
I20260929 18:32:06.621193 31148 raft_consensus.cc:2229] T f2af830f0afc4386a217d3bdd0a3abf0 P 28f15042e6c541e4b27b316554dbddfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:32:06.621338 31148 raft_consensus.cc:2258] T f2af830f0afc4386a217d3bdd0a3abf0 P 28f15042e6c541e4b27b316554dbddfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:06.621380 30893 tablet_service.cc:1514] Processing DeleteTablet for tablet f2af830f0afc4386a217d3bdd0a3abf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:06 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:34022
I20260929 18:32:06.621668 31149 tablet_replica.cc:330] T f2af830f0afc4386a217d3bdd0a3abf0 P 3c52aae6cbe94546b373509a42d6ca54: stopping tablet replica
I20260929 18:32:06.621778 31033 tablet_service.cc:1514] Processing DeleteTablet for tablet f2af830f0afc4386a217d3bdd0a3abf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:06 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:41792
I20260929 18:32:06.621824 31149 raft_consensus.cc:2229] T f2af830f0afc4386a217d3bdd0a3abf0 P 3c52aae6cbe94546b373509a42d6ca54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:32:06.621968 31149 raft_consensus.cc:2258] T f2af830f0afc4386a217d3bdd0a3abf0 P 3c52aae6cbe94546b373509a42d6ca54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:06.622054 31150 tablet_replica.cc:330] T f2af830f0afc4386a217d3bdd0a3abf0 P a4642f7baed54c09994b275f9837d000: stopping tablet replica
I20260929 18:32:06.622179 31150 raft_consensus.cc:2229] T f2af830f0afc4386a217d3bdd0a3abf0 P a4642f7baed54c09994b275f9837d000 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:32:06.622680 31150 raft_consensus.cc:2258] T f2af830f0afc4386a217d3bdd0a3abf0 P a4642f7baed54c09994b275f9837d000 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:06.623065 31149 ts_tablet_manager.cc:1905] T f2af830f0afc4386a217d3bdd0a3abf0 P 3c52aae6cbe94546b373509a42d6ca54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:32:06.624004 31148 ts_tablet_manager.cc:1905] T f2af830f0afc4386a217d3bdd0a3abf0 P 28f15042e6c541e4b27b316554dbddfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:32:06.627187 31150 ts_tablet_manager.cc:1905] T f2af830f0afc4386a217d3bdd0a3abf0 P a4642f7baed54c09994b275f9837d000: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:32:06.627777 31149 ts_tablet_manager.cc:1918] T f2af830f0afc4386a217d3bdd0a3abf0 P 3c52aae6cbe94546b373509a42d6ca54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:32:06.627777 31148 ts_tablet_manager.cc:1918] T f2af830f0afc4386a217d3bdd0a3abf0 P 28f15042e6c541e4b27b316554dbddfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:32:06.627872 31149 log.cc:1196] T f2af830f0afc4386a217d3bdd0a3abf0 P 3c52aae6cbe94546b373509a42d6ca54: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-1/wal/wals/f2af830f0afc4386a217d3bdd0a3abf0
I20260929 18:32:06.627880 31148 log.cc:1196] T f2af830f0afc4386a217d3bdd0a3abf0 P 28f15042e6c541e4b27b316554dbddfd: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-0/wal/wals/f2af830f0afc4386a217d3bdd0a3abf0
I20260929 18:32:06.628161 31148 ts_tablet_manager.cc:1939] T f2af830f0afc4386a217d3bdd0a3abf0 P 28f15042e6c541e4b27b316554dbddfd: Deleting consensus metadata
I20260929 18:32:06.628162 31149 ts_tablet_manager.cc:1939] T f2af830f0afc4386a217d3bdd0a3abf0 P 3c52aae6cbe94546b373509a42d6ca54: Deleting consensus metadata
I20260929 18:32:06.629709 30602 catalog_manager.cc:4841] TS 28f15042e6c541e4b27b316554dbddfd (127.28.207.65:44095): tablet f2af830f0afc4386a217d3bdd0a3abf0 (table test-table [id=2f72ccf2e9a243f3937693a663bf04d0]) successfully deleted
I20260929 18:32:06.629854 30601 catalog_manager.cc:4841] TS 3c52aae6cbe94546b373509a42d6ca54 (127.28.207.66:38011): tablet f2af830f0afc4386a217d3bdd0a3abf0 (table test-table [id=2f72ccf2e9a243f3937693a663bf04d0]) successfully deleted
I20260929 18:32:06.631438 31150 ts_tablet_manager.cc:1918] T f2af830f0afc4386a217d3bdd0a3abf0 P a4642f7baed54c09994b275f9837d000: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:32:06.631560 31150 log.cc:1196] T f2af830f0afc4386a217d3bdd0a3abf0 P a4642f7baed54c09994b275f9837d000: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790706718260899-29501-0/minicluster-data/ts-2/wal/wals/f2af830f0afc4386a217d3bdd0a3abf0
I20260929 18:32:06.631862 31150 ts_tablet_manager.cc:1939] T f2af830f0afc4386a217d3bdd0a3abf0 P a4642f7baed54c09994b275f9837d000: Deleting consensus metadata
I20260929 18:32:06.632885 30601 catalog_manager.cc:4841] TS a4642f7baed54c09994b275f9837d000 (127.28.207.67:37409): tablet f2af830f0afc4386a217d3bdd0a3abf0 (table test-table [id=2f72ccf2e9a243f3937693a663bf04d0]) successfully deleted
Sep 29 18:32:06 dist-test-slave-qcpv krb5kdc[30571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706725, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.207.65@KRBTEST.COM
I20260929 18:32:06.642473 31118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:06.623974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50444 (local address 127.28.207.65:44095)
0929 18:32:06.624113 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:06.624118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:06.628711 (+ 4593us) server_negotiation.cc:407] Connection header received
0929 18:32:06.629771 (+ 1060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:06.629775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:06.629834 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:06.629892 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:06.630885 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.631451 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.632439 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.632755 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.632857 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:06.636426 (+ 3569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:06.636441 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:06.636461 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:06.636494 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:06.639828 (+ 3334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:06.640440 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:06.640448 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:06.640450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:06.640549 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:06.641150 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:06.641155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:06.641157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:06.641533 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:06.641835 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:06.642089 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 18:32:06.642260 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260929 18:32:06.643131 30813 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50444
I20260929 18:32:06.655843 31161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:06.645707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41744 (local address 127.28.207.126:36751)
0929 18:32:06.645876 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:06.645879 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:06.645900 (+ 21us) server_negotiation.cc:407] Connection header received
0929 18:32:06.646042 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:06.646059 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:06.646124 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:06.646257 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:06.647177 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.648057 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.649018 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:06.649403 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:06.649548 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:06.651415 (+ 1867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:06.651420 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:06.651423 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:06.651460 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:06.653566 (+ 2106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:06.654161 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:06.654165 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:06.654167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:06.654285 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:06.654676 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:06.654680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:06.654682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:06.655014 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:06.655177 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:06.655485 (+ 308us) server_negotiation.cc:299] Negotiation successful
0929 18:32:06.655640 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260929 18:32:06.656344 30614 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41744
I20260929 18:32:06.658260 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 30688
I20260929 18:32:06.667667 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 30828
I20260929 18:32:06.676343 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 30968
I20260929 18:32:06.683898 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 30584
2026-09-29T18:32:06Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3334 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:32:06 dist-test-slave-qcpv krb5kdc[31189](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:32:06 dist-test-slave-qcpv krb5kdc[31189](info): set up 2 sockets
Sep 29 18:32:06 dist-test-slave-qcpv krb5kdc[31189](info): commencing operation
krb5kdc: starting...
W20260929 18:32:07.759548 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:32:07 dist-test-slave-qcpv krb5kdc[31189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:32:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:32:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.207.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
I20260929 18:32:07.907114 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:34381
--webserver_interface=127.28.207.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:38695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:34381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:08.048352 31202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:08.048857 31202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:08.048946 31202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:08.054672 31202 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:32:08.054838 31202 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 18:32:08.054908 31202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:08.054960 31202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:08.055174 31202 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:32:08.055287 31202 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:32:08.062233 31202 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:38695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:34381
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:34381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:08.063879 31202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:08.065270 31202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:08.076393 31208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:08.076426 31207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:08.076526 31202 server_base.cc:1034] running on GCE node
W20260929 18:32:08.076426 31210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:08.077548 31202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:08.078994 31202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:08.080282 31202 hybrid_clock.cc:648] HybridClock initialized: now 1790706728080247 us; error 59 us; skew 500 ppm
Sep 29 18:32:08 dist-test-slave-qcpv krb5kdc[31189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706728, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:08.083735 31202 init.cc:377] Logged in from keytab as kudu/127.28.207.126@KRBTEST.COM (short username kudu)
I20260929 18:32:08.084697 31202 webserver.cc:460] Webserver started at http://127.28.207.126:41599/ using document root <none> and password file <none>
I20260929 18:32:08.085255 31202 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:08.085323 31202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:08.085498 31202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:08.088281 31202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "6d27777167194a44a0b5510f5b8d6c99"
format_stamp: "Formatted at 2026-09-29 18:32:08 on dist-test-slave-qcpv"
server_key: "8a1eb0b640e4fbd6999b5928a721345e"
server_key_iv: "b729022718618f61496a660965c07000"
server_key_version: "encryptionkey@0"
I20260929 18:32:08.088948 31202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "6d27777167194a44a0b5510f5b8d6c99"
format_stamp: "Formatted at 2026-09-29 18:32:08 on dist-test-slave-qcpv"
server_key: "8a1eb0b640e4fbd6999b5928a721345e"
server_key_iv: "b729022718618f61496a660965c07000"
server_key_version: "encryptionkey@0"
I20260929 18:32:08.093751 31202 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 18:32:08.097316 31217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:08.099299 31202 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:32:08.099617 31202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "6d27777167194a44a0b5510f5b8d6c99"
format_stamp: "Formatted at 2026-09-29 18:32:08 on dist-test-slave-qcpv"
server_key: "8a1eb0b640e4fbd6999b5928a721345e"
server_key_iv: "b729022718618f61496a660965c07000"
server_key_version: "encryptionkey@0"
I20260929 18:32:08.099792 31202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:08.123535 31202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:08.128417 31202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:08.128714 31202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:08.140512 31202 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:34381
I20260929 18:32:08.142081 31202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:32:08.143082 31279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:34381 every 8 connection(s)
I20260929 18:32:08.145934 31280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:08.146080 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 31202
I20260929 18:32:08.146173 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:32:08.146484 29501 external_mini_cluster.cc:1428] Setting key a0349a9c6aced1fcb3b173028d0b1e74
I20260929 18:32:08.152967 31280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d27777167194a44a0b5510f5b8d6c99: Bootstrap starting.
I20260929 18:32:08.155925 31280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d27777167194a44a0b5510f5b8d6c99: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:08.156972 31280 log.cc:824] T 00000000000000000000000000000000 P 6d27777167194a44a0b5510f5b8d6c99: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:32:08 dist-test-slave-qcpv krb5kdc[31189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:32:08.160080 31280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d27777167194a44a0b5510f5b8d6c99: No bootstrap required, opened a new log
I20260929 18:32:08.163347 31280 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d27777167194a44a0b5510f5b8d6c99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d27777167194a44a0b5510f5b8d6c99" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 34381 } }
I20260929 18:32:08.163656 31280 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d27777167194a44a0b5510f5b8d6c99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:08.163735 31280 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d27777167194a44a0b5510f5b8d6c99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d27777167194a44a0b5510f5b8d6c99, State: Initialized, Role: FOLLOWER
I20260929 18:32:08.164203 31280 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d27777167194a44a0b5510f5b8d6c99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d27777167194a44a0b5510f5b8d6c99" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 34381 } }
I20260929 18:32:08.164373 31280 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d27777167194a44a0b5510f5b8d6c99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:32:08.164465 31280 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d27777167194a44a0b5510f5b8d6c99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:32:08.164588 31280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d27777167194a44a0b5510f5b8d6c99 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:08.165645 31280 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d27777167194a44a0b5510f5b8d6c99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d27777167194a44a0b5510f5b8d6c99" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 34381 } }
I20260929 18:32:08.166050 31280 leader_election.cc:302] T 00000000000000000000000000000000 P 6d27777167194a44a0b5510f5b8d6c99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d27777167194a44a0b5510f5b8d6c99; no voters:
I20260929 18:32:08.166441 31280 leader_election.cc:288] T 00000000000000000000000000000000 P 6d27777167194a44a0b5510f5b8d6c99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:32:08.166543 31285 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d27777167194a44a0b5510f5b8d6c99 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:32:08.166744 31285 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d27777167194a44a0b5510f5b8d6c99 [term 1 LEADER]: Becoming Leader. State: Replica: 6d27777167194a44a0b5510f5b8d6c99, State: Running, Role: LEADER
I20260929 18:32:08.167124 31285 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d27777167194a44a0b5510f5b8d6c99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d27777167194a44a0b5510f5b8d6c99" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 34381 } }
I20260929 18:32:08.167452 31283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:08.148498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44544 (local address 127.28.207.126:34381)
0929 18:32:08.149042 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:08.149067 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:08.149128 (+ 61us) server_negotiation.cc:407] Connection header received
0929 18:32:08.149974 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:08.149995 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:08.150441 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:08.151134 (+ 693us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:08.151999 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:08.153411 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:08.154734 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:08.155451 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:08.155628 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:08.159737 (+ 4109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:08.159832 (+ 95us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:08.159858 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:08.159904 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:08.163006 (+ 3102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:08.163637 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:08.163644 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:08.163651 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:08.163857 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:08.164239 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:08.164243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:08.164245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:08.164954 (+ 709us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:08.165226 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:08.165500 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 18:32:08.165944 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":187,"threads_started":1}
I20260929 18:32:08.167999 31280 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d27777167194a44a0b5510f5b8d6c99 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:32:08.170258 31286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d27777167194a44a0b5510f5b8d6c99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d27777167194a44a0b5510f5b8d6c99. Latest consensus state: current_term: 1 leader_uuid: "6d27777167194a44a0b5510f5b8d6c99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d27777167194a44a0b5510f5b8d6c99" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 34381 } } }
I20260929 18:32:08.170641 31287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d27777167194a44a0b5510f5b8d6c99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d27777167194a44a0b5510f5b8d6c99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d27777167194a44a0b5510f5b8d6c99" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 34381 } } }
I20260929 18:32:08.170815 31287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d27777167194a44a0b5510f5b8d6c99 [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:08.170841 31286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d27777167194a44a0b5510f5b8d6c99 [sys.catalog]: This master's current role is: LEADER
W20260929 18:32:08.173628 31299 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6d27777167194a44a0b5510f5b8d6c99: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:32:08.173720 31299 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:32:08.173898 31300 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:32:08.174989 31300 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:32:08.181737 31300 catalog_manager.cc:1347] Generated new cluster ID: f9266ab2323c4c22ab64dab71c61ccb4
I20260929 18:32:08.181960 31300 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:32:08.197603 31300 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:32:08.199028 31300 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:32:08.207883 31300 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d27777167194a44a0b5510f5b8d6c99: Generated new TSK 0
I20260929 18:32:08.208961 31300 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.207.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab.
Entry for principal kudu/127.28.207.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab.
I20260929 18:32:08.254761 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.65:0
--local_ip_for_outbound_sockets=127.28.207.65
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:34381
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:38695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:08.408667 31306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:08.409030 31306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:08.409086 31306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:08.413825 31306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:08.413916 31306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:08.413965 31306 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 18:32:08.413980 31306 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:32:08.414074 31306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.65
I20260929 18:32:08.420389 31306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:38695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.207.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.207.126:34381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.65
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:08.421900 31306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:08.423244 31306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:08.436184 31311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:08.436182 31312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:08.436184 31314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:08.436465 31306 server_base.cc:1034] running on GCE node
I20260929 18:32:08.436983 31306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:08.437757 31306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:08.439039 31306 hybrid_clock.cc:648] HybridClock initialized: now 1790706728439031 us; error 40 us; skew 500 ppm
Sep 29 18:32:08 dist-test-slave-qcpv krb5kdc[31189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706728, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:08.448992 31306 init.cc:377] Logged in from keytab as kudu/127.28.207.65@KRBTEST.COM (short username kudu)
I20260929 18:32:08.450150 31306 webserver.cc:460] Webserver started at http://127.28.207.65:44903/ using document root <none> and password file <none>
I20260929 18:32:08.450789 31306 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:08.450905 31306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:08.451152 31306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:08.453295 31306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-0/data/instance:
uuid: "1b4afce4018d4a93b66adbfd4b1f5e49"
format_stamp: "Formatted at 2026-09-29 18:32:08 on dist-test-slave-qcpv"
server_key: "009b5cee340da9054e4df8f328820eb0"
server_key_iv: "979c22f38954f2fa6d982f20f766884e"
server_key_version: "encryptionkey@0"
I20260929 18:32:08.453876 31306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance:
uuid: "1b4afce4018d4a93b66adbfd4b1f5e49"
format_stamp: "Formatted at 2026-09-29 18:32:08 on dist-test-slave-qcpv"
server_key: "009b5cee340da9054e4df8f328820eb0"
server_key_iv: "979c22f38954f2fa6d982f20f766884e"
server_key_version: "encryptionkey@0"
I20260929 18:32:08.458267 31306 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260929 18:32:08.461700 31321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:08.463536 31306 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 18:32:08.463718 31306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-0/wal
uuid: "1b4afce4018d4a93b66adbfd4b1f5e49"
format_stamp: "Formatted at 2026-09-29 18:32:08 on dist-test-slave-qcpv"
server_key: "009b5cee340da9054e4df8f328820eb0"
server_key_iv: "979c22f38954f2fa6d982f20f766884e"
server_key_version: "encryptionkey@0"
I20260929 18:32:08.463874 31306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:08.474267 31306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:08.478034 31306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:08.478258 31306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:08.480027 31306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:08.480103 31306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:08.480156 31306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:08.480177 31306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:08.492051 31306 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.65:46387
I20260929 18:32:08.492133 31436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.65:46387 every 8 connection(s)
Sep 29 18:32:08 dist-test-slave-qcpv krb5kdc[31189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706728, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:32:08.493098 31306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
I20260929 18:32:08.495603 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 31306
I20260929 18:32:08.495863 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance
I20260929 18:32:08.496366 29501 external_mini_cluster.cc:1428] Setting key 2ab176c41e27832f6467d2d902a8249a
I20260929 18:32:08.499940 31283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:08.481042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:47589 (local address 127.28.207.126:34381)
0929 18:32:08.481204 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:08.481208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:08.482397 (+ 1189us) server_negotiation.cc:407] Connection header received
0929 18:32:08.484174 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:08.484178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:08.484254 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:08.484371 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:08.486433 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:08.487265 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:08.488982 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:08.489242 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:08.489367 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:08.493807 (+ 4440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:08.493811 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:08.493814 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:08.493850 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:08.496605 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:08.497487 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:08.497490 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:08.497491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:08.497584 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:08.497999 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:08.498005 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:08.498006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:08.498298 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:08.498418 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:08.499666 (+ 1248us) server_negotiation.cc:299] Negotiation successful
0929 18:32:08.499805 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 18:32:08.501672 31330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:08.481507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:34381 (local address 127.28.207.65:47589)
0929 18:32:08.482168 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:08.482243 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:08.483848 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:08.484620 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:08.484629 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:08.485324 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:08.486224 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:08.486246 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:08.487412 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:08.487416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:08.488845 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:08.488856 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:08.489428 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:08.489432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:08.489694 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:08.490632 (+ 938us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:08.490682 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:08.493507 (+ 2825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:08.496812 (+ 3305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:08.496821 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:08.496842 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:08.497330 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:08.497705 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:08.497712 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:08.497715 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:08.497890 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:08.498493 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:08.498501 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:08.499324 (+ 823us) client_negotiation.cc:769] Sending connection context
0929 18:32:08.499627 (+ 303us) client_negotiation.cc:240] Negotiation successful
0929 18:32:08.500150 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":146,"threads_started":1}
I20260929 18:32:08.503803 31437 heartbeater.cc:347] Connected to a master server at 127.28.207.126:34381
I20260929 18:32:08.503989 31437 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:08.504736 31437 heartbeater.cc:511] Master 127.28.207.126:34381 requested a full tablet report, sending...
I20260929 18:32:08.507148 31234 ts_manager.cc:194] Registered new tserver with Master: 1b4afce4018d4a93b66adbfd4b1f5e49 (127.28.207.65:46387)
I20260929 18:32:08.509109 31234 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.65@KRBTEST.COM'} at 127.28.207.65:47589
WARNING: no policy specified for kudu/127.28.207.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.66@KRBTEST.COM" created.
I20260929 18:32:08.514906 31283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:08.506445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:42385 (local address 127.28.207.126:34381)
0929 18:32:08.506574 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:08.506578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:08.506810 (+ 232us) server_negotiation.cc:407] Connection header received
0929 18:32:08.507008 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:08.507011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:08.507062 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:08.507148 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:08.507646 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:08.508191 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:08.509032 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:08.509316 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:08.509408 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:08.510932 (+ 1524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:08.510936 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:08.510939 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:08.510968 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:08.512672 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:08.513332 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:08.513335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:08.513337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:08.513454 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:08.513788 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:08.513791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:08.513793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:08.514060 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:08.514193 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:08.514486 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 18:32:08.514657 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 18:32:08.514933 31444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:08.506402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:34381 (local address 127.28.207.65:42385)
0929 18:32:08.506716 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:08.506728 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:08.506901 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:08.507166 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:08.507168 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:08.507420 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:08.507534 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:08.507540 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:08.508314 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:08.508320 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:08.508915 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:08.508922 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:08.509418 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:08.509422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:08.509513 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:08.510103 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:08.510121 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:08.510709 (+ 588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:08.512846 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:08.512849 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:08.512852 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:08.513142 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:08.513580 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:08.513583 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:08.513586 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:08.513682 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:08.514177 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:08.514179 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:08.514297 (+ 118us) client_negotiation.cc:769] Sending connection context
0929 18:32:08.514392 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 18:32:08.514665 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":110,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab.
Entry for principal kudu/127.28.207.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab.
I20260929 18:32:08.534678 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.66:0
--local_ip_for_outbound_sockets=127.28.207.66
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:34381
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:38695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:08.686146 31446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:08.686450 31446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:08.686504 31446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:08.691919 31446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:08.692045 31446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:08.692097 31446 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 18:32:08.692113 31446 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:32:08.692181 31446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.66
I20260929 18:32:08.702252 31446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:38695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.207.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.207.126:34381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.66
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:08.704169 31446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:08.705673 31446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:08.718894 31452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:08.718875 31454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:08.719286 31446 server_base.cc:1034] running on GCE node
W20260929 18:32:08.718907 31451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:08.719863 31446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:08.720609 31446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:08.721853 31446 hybrid_clock.cc:648] HybridClock initialized: now 1790706728721815 us; error 38 us; skew 500 ppm
Sep 29 18:32:08 dist-test-slave-qcpv krb5kdc[31189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706728, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:08.725653 31446 init.cc:377] Logged in from keytab as kudu/127.28.207.66@KRBTEST.COM (short username kudu)
I20260929 18:32:08.727031 31446 webserver.cc:460] Webserver started at http://127.28.207.66:35015/ using document root <none> and password file <none>
I20260929 18:32:08.727768 31446 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:08.727846 31446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:08.728062 31446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:08.730913 31446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-1/data/instance:
uuid: "e2b2ec819b274e2a94fdeca8a004a587"
format_stamp: "Formatted at 2026-09-29 18:32:08 on dist-test-slave-qcpv"
server_key: "00961cef4517e80c1b18785a9e652ba2"
server_key_iv: "7fc08ece48ab756899fbbc3d3fd9feb1"
server_key_version: "encryptionkey@0"
I20260929 18:32:08.731596 31446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance:
uuid: "e2b2ec819b274e2a94fdeca8a004a587"
format_stamp: "Formatted at 2026-09-29 18:32:08 on dist-test-slave-qcpv"
server_key: "00961cef4517e80c1b18785a9e652ba2"
server_key_iv: "7fc08ece48ab756899fbbc3d3fd9feb1"
server_key_version: "encryptionkey@0"
I20260929 18:32:08.737299 31446 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 18:32:08.740785 31461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:08.742977 31446 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:32:08.743160 31446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-1/wal
uuid: "e2b2ec819b274e2a94fdeca8a004a587"
format_stamp: "Formatted at 2026-09-29 18:32:08 on dist-test-slave-qcpv"
server_key: "00961cef4517e80c1b18785a9e652ba2"
server_key_iv: "7fc08ece48ab756899fbbc3d3fd9feb1"
server_key_version: "encryptionkey@0"
I20260929 18:32:08.743316 31446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:08.770236 31446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:08.775077 31446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:08.775300 31446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:08.777442 31446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:08.777537 31446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:08.777661 31446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:08.777701 31446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:32:08 dist-test-slave-qcpv krb5kdc[31189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706728, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:32:08.792555 31446 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.66:41597
I20260929 18:32:08.792600 31576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.66:41597 every 8 connection(s)
I20260929 18:32:08.794196 31446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
I20260929 18:32:08.794945 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 31446
I20260929 18:32:08.795058 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance
I20260929 18:32:08.795380 29501 external_mini_cluster.cc:1428] Setting key 2abc36c56f3dc22631325270b44f0188
I20260929 18:32:08.796378 31283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:08.778305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:46055 (local address 127.28.207.126:34381)
0929 18:32:08.778457 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:08.778461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:08.780384 (+ 1923us) server_negotiation.cc:407] Connection header received
0929 18:32:08.781712 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:08.781716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:08.781781 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:08.781882 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:08.784249 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:08.784980 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:08.786329 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:08.786615 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:08.786738 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:08.790342 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:08.790345 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:08.790347 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:08.790373 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:08.792543 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:08.793412 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:08.793417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:08.793418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:08.793552 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:08.794167 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:08.794175 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:08.794176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:08.794456 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:08.794552 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:08.796110 (+ 1558us) server_negotiation.cc:299] Negotiation successful
0929 18:32:08.796255 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:32:08.796964 31470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:08.779499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:34381 (local address 127.28.207.66:46055)
0929 18:32:08.780188 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:08.780239 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:08.781472 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:08.782126 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:08.782138 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:08.782723 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:08.783961 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:08.783999 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:08.785151 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:08.785159 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:08.786179 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:08.786190 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:08.786798 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:08.786802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:08.787029 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:08.787803 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:08.787833 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:08.790131 (+ 2298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:08.792778 (+ 2647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:08.792785 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:08.792807 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:08.793229 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:08.793690 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:08.793694 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:08.793696 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:08.793907 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:08.794578 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:08.794588 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:08.795068 (+ 480us) client_negotiation.cc:769] Sending connection context
0929 18:32:08.795314 (+ 246us) client_negotiation.cc:240] Negotiation successful
0929 18:32:08.795740 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":190,"threads_started":1}
I20260929 18:32:08.800014 31577 heartbeater.cc:347] Connected to a master server at 127.28.207.126:34381
I20260929 18:32:08.800256 31577 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:08.801084 31577 heartbeater.cc:511] Master 127.28.207.126:34381 requested a full tablet report, sending...
I20260929 18:32:08.802628 31233 ts_manager.cc:194] Registered new tserver with Master: e2b2ec819b274e2a94fdeca8a004a587 (127.28.207.66:41597)
I20260929 18:32:08.803889 31233 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.66@KRBTEST.COM'} at 127.28.207.66:46055
WARNING: no policy specified for kudu/127.28.207.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.67@KRBTEST.COM" created.
I20260929 18:32:08.818123 31283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:08.808426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:58235 (local address 127.28.207.126:34381)
0929 18:32:08.808675 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:08.808681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:08.808940 (+ 259us) server_negotiation.cc:407] Connection header received
0929 18:32:08.809093 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:08.809096 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:08.809151 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:08.809244 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:08.809888 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:08.810549 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:08.811938 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:08.812197 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:08.812332 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:08.814190 (+ 1858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:08.814194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:08.814196 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:08.814231 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:08.815963 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:08.816749 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:08.816753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:08.816754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:08.816833 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:08.817155 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:08.817157 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:08.817159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:08.817408 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:08.817487 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:08.817856 (+ 369us) server_negotiation.cc:299] Negotiation successful
0929 18:32:08.817970 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260929 18:32:08.818310 31584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:08.808261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:34381 (local address 127.28.207.66:58235)
0929 18:32:08.808731 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:08.808749 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:08.808958 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:08.809325 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:08.809327 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:08.809579 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:08.809709 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:08.809716 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:08.810792 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:08.810799 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:08.811758 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:08.811770 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:08.812353 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:08.812358 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:08.812465 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:08.813205 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:08.813226 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:08.813967 (+ 741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:08.816116 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:08.816124 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:08.816129 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:08.816498 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:08.816941 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:08.816944 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:08.816946 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:08.817044 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:08.817514 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:08.817519 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:08.817666 (+ 147us) client_negotiation.cc:769] Sending connection context
0929 18:32:08.817797 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 18:32:08.817998 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":159,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab.
Entry for principal kudu/127.28.207.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab.
I20260929 18:32:08.831811 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.67:0
--local_ip_for_outbound_sockets=127.28.207.67
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:34381
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:38695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:08.983042 31586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:08.983474 31586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:08.983569 31586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:08.988441 31586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:08.988587 31586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:08.988677 31586 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 18:32:08.988725 31586 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:32:08.988795 31586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.67
I20260929 18:32:08.996815 31586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:38695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.207.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.207.126:34381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.67
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:08.999716 31586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:09.001319 31586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:09.014838 31592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:09.014931 31594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:09.014925 31591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:09.015247 31586 server_base.cc:1034] running on GCE node
I20260929 18:32:09.016219 31586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:09.017199 31586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:09.018503 31586 hybrid_clock.cc:648] HybridClock initialized: now 1790706729018465 us; error 47 us; skew 500 ppm
Sep 29 18:32:09 dist-test-slave-qcpv krb5kdc[31189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706729, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:09.022207 31586 init.cc:377] Logged in from keytab as kudu/127.28.207.67@KRBTEST.COM (short username kudu)
I20260929 18:32:09.023696 31586 webserver.cc:460] Webserver started at http://127.28.207.67:46173/ using document root <none> and password file <none>
I20260929 18:32:09.024494 31586 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:09.024618 31586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:09.024922 31586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:09.027865 31586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-2/data/instance:
uuid: "50a58713495847f580c2a4e37fc4fd2b"
format_stamp: "Formatted at 2026-09-29 18:32:09 on dist-test-slave-qcpv"
server_key: "db7b0c4adfb062107342292b7a69a0cd"
server_key_iv: "2f1cf7a4eb1f13f9b6c9f0193df4f031"
server_key_version: "encryptionkey@0"
I20260929 18:32:09.028643 31586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance:
uuid: "50a58713495847f580c2a4e37fc4fd2b"
format_stamp: "Formatted at 2026-09-29 18:32:09 on dist-test-slave-qcpv"
server_key: "db7b0c4adfb062107342292b7a69a0cd"
server_key_iv: "2f1cf7a4eb1f13f9b6c9f0193df4f031"
server_key_version: "encryptionkey@0"
I20260929 18:32:09.034706 31586 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 18:32:09.038448 31601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:09.040333 31586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:32:09.040560 31586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-2/wal
uuid: "50a58713495847f580c2a4e37fc4fd2b"
format_stamp: "Formatted at 2026-09-29 18:32:09 on dist-test-slave-qcpv"
server_key: "db7b0c4adfb062107342292b7a69a0cd"
server_key_iv: "2f1cf7a4eb1f13f9b6c9f0193df4f031"
server_key_version: "encryptionkey@0"
I20260929 18:32:09.040761 31586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:09.057785 31586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:09.064520 31586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:09.064812 31586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:09.066810 31586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:09.066934 31586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260929 18:32:09.067035 31586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:09.067090 31586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:32:09 dist-test-slave-qcpv krb5kdc[31189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706729, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:32:09.083020 31586 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.67:34053
I20260929 18:32:09.083372 31716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.67:34053 every 8 connection(s)
I20260929 18:32:09.084430 31586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
I20260929 18:32:09.086299 31283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:09.068555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:46597 (local address 127.28.207.126:34381)
0929 18:32:09.068710 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:09.068715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:09.070286 (+ 1571us) server_negotiation.cc:407] Connection header received
0929 18:32:09.072016 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:09.072020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:09.072086 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:09.072196 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:09.075107 (+ 2911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.075713 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.077178 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.077408 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.077534 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:09.081629 (+ 4095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:09.081632 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:09.081635 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:09.081663 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:09.083827 (+ 2164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:09.084380 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:09.084385 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:09.084387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:09.084491 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:09.084860 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:09.084866 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:09.084867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:09.085134 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:09.085228 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:09.085980 (+ 752us) server_negotiation.cc:299] Negotiation successful
0929 18:32:09.086145 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:32:09.087282 31610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:09.069217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:34381 (local address 127.28.207.67:46597)
0929 18:32:09.069994 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:09.070054 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:09.071732 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:09.072497 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:09.072513 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:09.073244 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:09.074627 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:09.074662 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.075893 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.075898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:09.077021 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:09.077034 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.077602 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.077607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:09.077791 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:09.078550 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:09.078568 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:09.081374 (+ 2806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:09.083958 (+ 2584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:09.083965 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:09.083985 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:09.084267 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:09.084604 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:09.084607 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:09.084608 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:09.084756 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:09.085281 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:09.085288 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:09.085607 (+ 319us) client_negotiation.cc:769] Sending connection context
0929 18:32:09.085843 (+ 236us) client_negotiation.cc:240] Negotiation successful
0929 18:32:09.086196 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":170,"threads_started":1}
I20260929 18:32:09.088896 31717 heartbeater.cc:347] Connected to a master server at 127.28.207.126:34381
I20260929 18:32:09.089172 31717 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:09.089860 31717 heartbeater.cc:511] Master 127.28.207.126:34381 requested a full tablet report, sending...
I20260929 18:32:09.091415 31234 ts_manager.cc:194] Registered new tserver with Master: 50a58713495847f580c2a4e37fc4fd2b (127.28.207.67:34053)
I20260929 18:32:09.091995 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 31586
I20260929 18:32:09.092123 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance
I20260929 18:32:09.092329 31234 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.67@KRBTEST.COM'} at 127.28.207.67:46597
I20260929 18:32:09.092492 29501 external_mini_cluster.cc:1428] Setting key f1512660f59a483a5968030150438ae7
I20260929 18:32:09.097026 29501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:32:09.102102 31723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:09.091886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:34381 (local address 127.28.207.67:34149)
0929 18:32:09.092231 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:09.092246 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:09.092362 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:09.093366 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:09.093371 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:09.093633 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:09.093777 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:09.093787 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.094710 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.094713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:09.095580 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:09.095590 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.096232 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.096235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:09.096369 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:09.097058 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:09.097069 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:09.097464 (+ 395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:09.100041 (+ 2577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:09.100047 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:09.100051 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:09.100373 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:09.100826 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:09.100831 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:09.100835 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:09.100955 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:09.101504 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:09.101507 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:09.101611 (+ 104us) client_negotiation.cc:769] Sending connection context
0929 18:32:09.101738 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 18:32:09.101901 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":144,"threads_started":1}
I20260929 18:32:09.102103 31283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:09.092885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:34149 (local address 127.28.207.126:34381)
0929 18:32:09.093058 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:09.093064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:09.093083 (+ 19us) server_negotiation.cc:407] Connection header received
0929 18:32:09.093132 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:09.093136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:09.093193 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:09.093311 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:09.093969 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.094549 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.095749 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.096086 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.096186 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:09.097638 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:09.097642 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:09.097645 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:09.097676 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:09.099863 (+ 2187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:09.100594 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:09.100597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:09.100598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:09.100685 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:09.101130 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:09.101134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:09.101135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:09.101380 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:09.101484 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:09.101806 (+ 322us) server_negotiation.cc:299] Negotiation successful
0929 18:32:09.101955 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
Sep 29 18:32:09 dist-test-slave-qcpv krb5kdc[31189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 18:32:09 dist-test-slave-qcpv krb5kdc[31189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:32:09.129727 31283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:09.120159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44550 (local address 127.28.207.126:34381)
0929 18:32:09.120347 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:09.120351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:09.120367 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:32:09.120411 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:09.120413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:09.120461 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:09.120556 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:09.121329 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.121904 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.122739 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.123065 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.123174 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:09.125690 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:09.125694 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:09.125697 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:09.125732 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:09.127944 (+ 2212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:09.128458 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:09.128463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:09.128464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:09.128540 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:09.128825 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:09.128828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:09.128830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:09.129024 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:09.129162 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:09.129433 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 18:32:09.129557 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 18:32:09.133754 31234 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:32:09.137173 31234 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:32:09.157828 31735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:09.149351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:34053 (local address 127.0.0.1:52934)
0929 18:32:09.151154 (+ 1803us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:09.151218 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:09.151352 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:09.152075 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:09.152079 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:09.152112 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:09.152273 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:09.152281 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.154046 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.154051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:09.155721 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:09.155747 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.157144 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.157149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:09.157308 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:09.157391 (+ 83us) client_negotiation.cc:769] Sending connection context
0929 18:32:09.157511 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 18:32:09.157648 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1725,"mutex_wait_us":26,"spinlock_wait_cycles":5504,"thread_start_us":147,"threads_started":1}
I20260929 18:32:09.158594 31739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:09.150734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43920 (local address 127.28.207.65:46387)
0929 18:32:09.151744 (+ 1010us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:09.151750 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:09.152000 (+ 250us) server_negotiation.cc:407] Connection header received
0929 18:32:09.152196 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:09.152201 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:09.152517 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:09.152670 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:09.153158 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.154165 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.156621 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.157504 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.157615 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:09.157801 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:09.158180 (+ 379us) server_negotiation.cc:299] Negotiation successful
0929 18:32:09.158449 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":922,"thread_start_us":171,"threads_started":1}
I20260929 18:32:09.159045 31738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:09.150604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52934 (local address 127.28.207.67:34053)
0929 18:32:09.150973 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:09.150987 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:09.151308 (+ 321us) server_negotiation.cc:407] Connection header received
0929 18:32:09.151497 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:09.151503 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:09.151744 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:09.151912 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:09.152414 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.153903 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.155905 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.156861 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.157001 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:09.157172 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:09.158661 (+ 1489us) server_negotiation.cc:299] Negotiation successful
0929 18:32:09.158926 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":118,"threads_started":1}
I20260929 18:32:09.159013 31737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:09.150176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:46387 (local address 127.0.0.1:43920)
0929 18:32:09.151914 (+ 1738us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:09.151930 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:09.152063 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:09.152870 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:09.152873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:09.152895 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:09.153011 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:09.153017 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.154306 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.154310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:09.156103 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:09.156115 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.157902 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.157905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:09.158007 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:09.158032 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 18:32:09.158115 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 18:32:09.158503 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1671,"thread_start_us":109,"threads_started":1}
I20260929 18:32:09.159950 31734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:09.149151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:41597 (local address 127.0.0.1:41982)
0929 18:32:09.151449 (+ 2298us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:09.151459 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:09.151546 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:09.152046 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:09.152052 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:09.152071 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:09.152264 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:09.152272 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.155355 (+ 3083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.155359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:09.157487 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:09.157497 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.159451 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.159455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:09.159575 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:09.159601 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 18:32:09.159669 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 18:32:09.159755 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1271,"mutex_wait_us":3,"thread_start_us":114,"threads_started":1}
I20260929 18:32:09.160497 31736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:09.149714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41982 (local address 127.28.207.66:41597)
0929 18:32:09.150195 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:09.150211 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:09.151556 (+ 1345us) server_negotiation.cc:407] Connection header received
0929 18:32:09.151647 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:09.151652 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:09.151857 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:09.152577 (+ 720us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:09.152766 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.155191 (+ 2425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.157672 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.159313 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.159453 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:09.159803 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:09.159952 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 18:32:09.160165 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":106,"threads_started":1}
I20260929 18:32:09.162290 31651 tablet_service.cc:1467] Processing CreateTablet for tablet b21c5b1e8621467ba7fb32400dabf70e (DEFAULT_TABLE table=test-table [id=e4932755fb2140dab385d6ad43f4a257]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:09.162289 31371 tablet_service.cc:1467] Processing CreateTablet for tablet b21c5b1e8621467ba7fb32400dabf70e (DEFAULT_TABLE table=test-table [id=e4932755fb2140dab385d6ad43f4a257]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:09.163503 31371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b21c5b1e8621467ba7fb32400dabf70e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:09.163507 31651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b21c5b1e8621467ba7fb32400dabf70e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:09.163950 31511 tablet_service.cc:1467] Processing CreateTablet for tablet b21c5b1e8621467ba7fb32400dabf70e (DEFAULT_TABLE table=test-table [id=e4932755fb2140dab385d6ad43f4a257]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:09.165259 31511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b21c5b1e8621467ba7fb32400dabf70e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:09.172384 31740 tablet_bootstrap.cc:492] T b21c5b1e8621467ba7fb32400dabf70e P 1b4afce4018d4a93b66adbfd4b1f5e49: Bootstrap starting.
I20260929 18:32:09.173257 31741 tablet_bootstrap.cc:492] T b21c5b1e8621467ba7fb32400dabf70e P 50a58713495847f580c2a4e37fc4fd2b: Bootstrap starting.
I20260929 18:32:09.174767 31742 tablet_bootstrap.cc:492] T b21c5b1e8621467ba7fb32400dabf70e P e2b2ec819b274e2a94fdeca8a004a587: Bootstrap starting.
I20260929 18:32:09.175992 31741 tablet_bootstrap.cc:654] T b21c5b1e8621467ba7fb32400dabf70e P 50a58713495847f580c2a4e37fc4fd2b: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:09.176587 31740 tablet_bootstrap.cc:654] T b21c5b1e8621467ba7fb32400dabf70e P 1b4afce4018d4a93b66adbfd4b1f5e49: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:09.176992 31741 log.cc:824] T b21c5b1e8621467ba7fb32400dabf70e P 50a58713495847f580c2a4e37fc4fd2b: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:09.177371 31742 tablet_bootstrap.cc:654] T b21c5b1e8621467ba7fb32400dabf70e P e2b2ec819b274e2a94fdeca8a004a587: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:09.178186 31740 log.cc:824] T b21c5b1e8621467ba7fb32400dabf70e P 1b4afce4018d4a93b66adbfd4b1f5e49: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:09.178259 31742 log.cc:824] T b21c5b1e8621467ba7fb32400dabf70e P e2b2ec819b274e2a94fdeca8a004a587: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:09.179710 31741 tablet_bootstrap.cc:492] T b21c5b1e8621467ba7fb32400dabf70e P 50a58713495847f580c2a4e37fc4fd2b: No bootstrap required, opened a new log
I20260929 18:32:09.179949 31741 ts_tablet_manager.cc:1397] T b21c5b1e8621467ba7fb32400dabf70e P 50a58713495847f580c2a4e37fc4fd2b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 18:32:09.180250 31740 tablet_bootstrap.cc:492] T b21c5b1e8621467ba7fb32400dabf70e P 1b4afce4018d4a93b66adbfd4b1f5e49: No bootstrap required, opened a new log
I20260929 18:32:09.180265 31742 tablet_bootstrap.cc:492] T b21c5b1e8621467ba7fb32400dabf70e P e2b2ec819b274e2a94fdeca8a004a587: No bootstrap required, opened a new log
I20260929 18:32:09.180444 31742 ts_tablet_manager.cc:1397] T b21c5b1e8621467ba7fb32400dabf70e P e2b2ec819b274e2a94fdeca8a004a587: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:32:09.180444 31740 ts_tablet_manager.cc:1397] T b21c5b1e8621467ba7fb32400dabf70e P 1b4afce4018d4a93b66adbfd4b1f5e49: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 18:32:09.184666 31741 raft_consensus.cc:348] T b21c5b1e8621467ba7fb32400dabf70e P 50a58713495847f580c2a4e37fc4fd2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } } peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } }
I20260929 18:32:09.185030 31740 raft_consensus.cc:348] T b21c5b1e8621467ba7fb32400dabf70e P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } } peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } }
I20260929 18:32:09.184922 31742 raft_consensus.cc:348] T b21c5b1e8621467ba7fb32400dabf70e P e2b2ec819b274e2a94fdeca8a004a587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } } peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } }
I20260929 18:32:09.185245 31740 raft_consensus.cc:374] T b21c5b1e8621467ba7fb32400dabf70e P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:09.185247 31742 raft_consensus.cc:374] T b21c5b1e8621467ba7fb32400dabf70e P e2b2ec819b274e2a94fdeca8a004a587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:09.185278 31741 raft_consensus.cc:374] T b21c5b1e8621467ba7fb32400dabf70e P 50a58713495847f580c2a4e37fc4fd2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:09.185323 31742 raft_consensus.cc:729] T b21c5b1e8621467ba7fb32400dabf70e P e2b2ec819b274e2a94fdeca8a004a587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2b2ec819b274e2a94fdeca8a004a587, State: Initialized, Role: FOLLOWER
I20260929 18:32:09.185359 31741 raft_consensus.cc:729] T b21c5b1e8621467ba7fb32400dabf70e P 50a58713495847f580c2a4e37fc4fd2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50a58713495847f580c2a4e37fc4fd2b, State: Initialized, Role: FOLLOWER
I20260929 18:32:09.185365 31740 raft_consensus.cc:729] T b21c5b1e8621467ba7fb32400dabf70e P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b4afce4018d4a93b66adbfd4b1f5e49, State: Initialized, Role: FOLLOWER
I20260929 18:32:09.185885 31740 consensus_queue.cc:260] T b21c5b1e8621467ba7fb32400dabf70e P 1b4afce4018d4a93b66adbfd4b1f5e49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } } peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } }
I20260929 18:32:09.185875 31742 consensus_queue.cc:260] T b21c5b1e8621467ba7fb32400dabf70e P e2b2ec819b274e2a94fdeca8a004a587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } } peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } }
I20260929 18:32:09.185937 31741 consensus_queue.cc:260] T b21c5b1e8621467ba7fb32400dabf70e P 50a58713495847f580c2a4e37fc4fd2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } } peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } }
I20260929 18:32:09.187031 31740 ts_tablet_manager.cc:1428] T b21c5b1e8621467ba7fb32400dabf70e P 1b4afce4018d4a93b66adbfd4b1f5e49: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260929 18:32:09.187031 31742 ts_tablet_manager.cc:1428] T b21c5b1e8621467ba7fb32400dabf70e P e2b2ec819b274e2a94fdeca8a004a587: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260929 18:32:09.187431 31741 ts_tablet_manager.cc:1428] T b21c5b1e8621467ba7fb32400dabf70e P 50a58713495847f580c2a4e37fc4fd2b: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260929 18:32:09.187423 31437 heartbeater.cc:503] Master 127.28.207.126:34381 was elected leader, sending a full tablet report...
I20260929 18:32:09.187860 31717 heartbeater.cc:503] Master 127.28.207.126:34381 was elected leader, sending a full tablet report...
I20260929 18:32:09.188103 31577 heartbeater.cc:503] Master 127.28.207.126:34381 was elected leader, sending a full tablet report...
W20260929 18:32:09.244626 31438 tablet.cc:2369] T b21c5b1e8621467ba7fb32400dabf70e P 1b4afce4018d4a93b66adbfd4b1f5e49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:32:09.285535 31748 raft_consensus.cc:482] T b21c5b1e8621467ba7fb32400dabf70e P 50a58713495847f580c2a4e37fc4fd2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:32:09.285890 31748 raft_consensus.cc:504] T b21c5b1e8621467ba7fb32400dabf70e P 50a58713495847f580c2a4e37fc4fd2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } } peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } }
I20260929 18:32:09.287637 31748 leader_election.cc:288] T b21c5b1e8621467ba7fb32400dabf70e P 50a58713495847f580c2a4e37fc4fd2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2b2ec819b274e2a94fdeca8a004a587 (127.28.207.66:41597), 1b4afce4018d4a93b66adbfd4b1f5e49 (127.28.207.65:46387)
I20260929 18:32:09.292578 31739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:09.288229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:33263 (local address 127.28.207.65:46387)
0929 18:32:09.288391 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:09.288396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:09.288414 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:32:09.288721 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:09.288725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:09.288809 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:09.288933 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:09.289331 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.289913 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.291305 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.291752 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.291964 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:09.292023 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:09.292321 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 18:32:09.292411 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 18:32:09.292761 31610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:09.288051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:46387 (local address 127.28.207.67:33263)
0929 18:32:09.288290 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:09.288311 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:09.288473 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:09.288962 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:09.288966 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:09.288986 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:09.289173 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:09.289207 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.290055 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.290058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:09.291126 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:09.291143 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.291913 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.291918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:09.292171 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:09.292200 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 18:32:09.292292 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 18:32:09.292564 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":26}
I20260929 18:32:09.293246 31736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:09.288292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:54337 (local address 127.28.207.66:41597)
0929 18:32:09.288447 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:09.288455 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:09.288904 (+ 449us) server_negotiation.cc:407] Connection header received
0929 18:32:09.289044 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:09.289049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:09.289103 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:09.289168 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:09.289991 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.290721 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.291953 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.292466 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.292595 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:09.292656 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:09.292987 (+ 331us) server_negotiation.cc:299] Negotiation successful
0929 18:32:09.293116 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 18:32:09.293679 31749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:09.288051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:41597 (local address 127.28.207.67:54337)
0929 18:32:09.288785 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:09.288798 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:09.288922 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:09.289203 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:09.289207 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:09.289222 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:09.289852 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:09.289860 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.290883 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.290886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:09.291773 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:09.291781 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.292637 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.292641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:09.292754 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:09.292795 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 18:32:09.293446 (+ 651us) client_negotiation.cc:240] Negotiation successful
0929 18:32:09.293527 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"mutex_wait_us":216,"thread_start_us":164,"threads_started":1}
I20260929 18:32:09.293742 31391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21c5b1e8621467ba7fb32400dabf70e" candidate_uuid: "50a58713495847f580c2a4e37fc4fd2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" is_pre_election: true
I20260929 18:32:09.294178 31391 raft_consensus.cc:2454] T b21c5b1e8621467ba7fb32400dabf70e P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50a58713495847f580c2a4e37fc4fd2b in term 0.
I20260929 18:32:09.294513 31524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21c5b1e8621467ba7fb32400dabf70e" candidate_uuid: "50a58713495847f580c2a4e37fc4fd2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2b2ec819b274e2a94fdeca8a004a587" is_pre_election: true
I20260929 18:32:09.294793 31604 leader_election.cc:302] T b21c5b1e8621467ba7fb32400dabf70e P 50a58713495847f580c2a4e37fc4fd2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b4afce4018d4a93b66adbfd4b1f5e49, 50a58713495847f580c2a4e37fc4fd2b; no voters:
I20260929 18:32:09.294943 31524 raft_consensus.cc:2454] T b21c5b1e8621467ba7fb32400dabf70e P e2b2ec819b274e2a94fdeca8a004a587 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50a58713495847f580c2a4e37fc4fd2b in term 0.
I20260929 18:32:09.295205 31748 raft_consensus.cc:2781] T b21c5b1e8621467ba7fb32400dabf70e P 50a58713495847f580c2a4e37fc4fd2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:32:09.295305 31748 raft_consensus.cc:482] T b21c5b1e8621467ba7fb32400dabf70e P 50a58713495847f580c2a4e37fc4fd2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:32:09.295338 31748 raft_consensus.cc:3037] T b21c5b1e8621467ba7fb32400dabf70e P 50a58713495847f580c2a4e37fc4fd2b [term 0 FOLLOWER]: Advancing to term 1
W20260929 18:32:09.295984 31578 tablet.cc:2369] T b21c5b1e8621467ba7fb32400dabf70e P e2b2ec819b274e2a94fdeca8a004a587: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:32:09.296896 31748 raft_consensus.cc:504] T b21c5b1e8621467ba7fb32400dabf70e P 50a58713495847f580c2a4e37fc4fd2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } } peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } }
I20260929 18:32:09.297392 31748 leader_election.cc:288] T b21c5b1e8621467ba7fb32400dabf70e P 50a58713495847f580c2a4e37fc4fd2b [CANDIDATE]: Term 1 election: Requested vote from peers e2b2ec819b274e2a94fdeca8a004a587 (127.28.207.66:41597), 1b4afce4018d4a93b66adbfd4b1f5e49 (127.28.207.65:46387)
I20260929 18:32:09.297935 31524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21c5b1e8621467ba7fb32400dabf70e" candidate_uuid: "50a58713495847f580c2a4e37fc4fd2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2b2ec819b274e2a94fdeca8a004a587"
I20260929 18:32:09.298085 31524 raft_consensus.cc:3037] T b21c5b1e8621467ba7fb32400dabf70e P e2b2ec819b274e2a94fdeca8a004a587 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:09.298022 31391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21c5b1e8621467ba7fb32400dabf70e" candidate_uuid: "50a58713495847f580c2a4e37fc4fd2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49"
I20260929 18:32:09.298313 31391 raft_consensus.cc:3037] T b21c5b1e8621467ba7fb32400dabf70e P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:09.299558 31524 raft_consensus.cc:2454] T b21c5b1e8621467ba7fb32400dabf70e P e2b2ec819b274e2a94fdeca8a004a587 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50a58713495847f580c2a4e37fc4fd2b in term 1.
I20260929 18:32:09.300031 31391 raft_consensus.cc:2454] T b21c5b1e8621467ba7fb32400dabf70e P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50a58713495847f580c2a4e37fc4fd2b in term 1.
I20260929 18:32:09.300076 31605 leader_election.cc:302] T b21c5b1e8621467ba7fb32400dabf70e P 50a58713495847f580c2a4e37fc4fd2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50a58713495847f580c2a4e37fc4fd2b, e2b2ec819b274e2a94fdeca8a004a587; no voters:
I20260929 18:32:09.300336 31748 raft_consensus.cc:2781] T b21c5b1e8621467ba7fb32400dabf70e P 50a58713495847f580c2a4e37fc4fd2b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:32:09.300637 31748 raft_consensus.cc:686] T b21c5b1e8621467ba7fb32400dabf70e P 50a58713495847f580c2a4e37fc4fd2b [term 1 LEADER]: Becoming Leader. State: Replica: 50a58713495847f580c2a4e37fc4fd2b, State: Running, Role: LEADER
I20260929 18:32:09.301193 31748 consensus_queue.cc:237] T b21c5b1e8621467ba7fb32400dabf70e P 50a58713495847f580c2a4e37fc4fd2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } } peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } }
I20260929 18:32:09.305749 31233 catalog_manager.cc:5495] T b21c5b1e8621467ba7fb32400dabf70e P 50a58713495847f580c2a4e37fc4fd2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 50a58713495847f580c2a4e37fc4fd2b (127.28.207.67). New cstate: current_term: 1 leader_uuid: "50a58713495847f580c2a4e37fc4fd2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } health_report { overall_health: UNKNOWN } } }
W20260929 18:32:09.335789 31718 tablet.cc:2369] T b21c5b1e8621467ba7fb32400dabf70e P 50a58713495847f580c2a4e37fc4fd2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 18:32:09 dist-test-slave-qcpv krb5kdc[31189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706728, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:32:09.362552 31757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:09.349553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:34381 (local address 127.0.0.1:44566)
0929 18:32:09.349929 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:09.349946 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:09.350100 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:09.350417 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:09.350420 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:09.350995 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:09.351120 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:09.351127 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.353434 (+ 2307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.353438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:09.354123 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:09.354132 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.354709 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.354713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:09.354790 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:09.355751 (+ 961us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:09.355767 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:09.358058 (+ 2291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:09.360540 (+ 2482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:09.360553 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:09.360561 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:09.360893 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:09.361296 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:09.361299 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:09.361301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:09.361403 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:09.361898 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:09.361907 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:09.362074 (+ 167us) client_negotiation.cc:769] Sending connection context
0929 18:32:09.362199 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 18:32:09.362368 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":118,"threads_started":1}
I20260929 18:32:09.362708 31283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:09.349776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44566 (local address 127.28.207.126:34381)
0929 18:32:09.349970 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:09.349974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:09.350098 (+ 124us) server_negotiation.cc:407] Connection header received
0929 18:32:09.350191 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:09.350193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:09.350257 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:09.350342 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:09.351279 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.352659 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.354325 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.354569 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.354674 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:09.358311 (+ 3637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:09.358315 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:09.358317 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:09.358358 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:09.360360 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:09.361037 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:09.361040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:09.361042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:09.361152 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:09.361503 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:09.361505 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:09.361506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:09.361720 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:09.361840 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:09.362335 (+ 495us) server_negotiation.cc:299] Negotiation successful
0929 18:32:09.362475 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 18:32:09.368347 31233 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:44566:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 18:32:09.369393 31233 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:32:09.375247 31371 tablet_service.cc:1467] Processing CreateTablet for tablet 13519206d3fc4377a65740d64de08342 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b359628a3c2a425f83a7c4c449c0af22]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:32:09.375227 31511 tablet_service.cc:1467] Processing CreateTablet for tablet 13519206d3fc4377a65740d64de08342 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b359628a3c2a425f83a7c4c449c0af22]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:32:09.376102 31371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13519206d3fc4377a65740d64de08342. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:09.376102 31511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13519206d3fc4377a65740d64de08342. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:09.375701 31651 tablet_service.cc:1467] Processing CreateTablet for tablet 13519206d3fc4377a65740d64de08342 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b359628a3c2a425f83a7c4c449c0af22]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:32:09.376781 31651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13519206d3fc4377a65740d64de08342. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:09.379673 31740 tablet_bootstrap.cc:492] T 13519206d3fc4377a65740d64de08342 P 1b4afce4018d4a93b66adbfd4b1f5e49: Bootstrap starting.
I20260929 18:32:09.381117 31740 tablet_bootstrap.cc:654] T 13519206d3fc4377a65740d64de08342 P 1b4afce4018d4a93b66adbfd4b1f5e49: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:09.382017 31742 tablet_bootstrap.cc:492] T 13519206d3fc4377a65740d64de08342 P e2b2ec819b274e2a94fdeca8a004a587: Bootstrap starting.
I20260929 18:32:09.382166 31741 tablet_bootstrap.cc:492] T 13519206d3fc4377a65740d64de08342 P 50a58713495847f580c2a4e37fc4fd2b: Bootstrap starting.
I20260929 18:32:09.383354 31741 tablet_bootstrap.cc:654] T 13519206d3fc4377a65740d64de08342 P 50a58713495847f580c2a4e37fc4fd2b: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:09.383354 31742 tablet_bootstrap.cc:654] T 13519206d3fc4377a65740d64de08342 P e2b2ec819b274e2a94fdeca8a004a587: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:09.383554 31740 tablet_bootstrap.cc:492] T 13519206d3fc4377a65740d64de08342 P 1b4afce4018d4a93b66adbfd4b1f5e49: No bootstrap required, opened a new log
I20260929 18:32:09.383708 31740 ts_tablet_manager.cc:1397] T 13519206d3fc4377a65740d64de08342 P 1b4afce4018d4a93b66adbfd4b1f5e49: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 18:32:09.384249 31740 raft_consensus.cc:348] T 13519206d3fc4377a65740d64de08342 P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } }
I20260929 18:32:09.384523 31740 raft_consensus.cc:374] T 13519206d3fc4377a65740d64de08342 P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:09.384584 31740 raft_consensus.cc:729] T 13519206d3fc4377a65740d64de08342 P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b4afce4018d4a93b66adbfd4b1f5e49, State: Initialized, Role: FOLLOWER
I20260929 18:32:09.384775 31740 consensus_queue.cc:260] T 13519206d3fc4377a65740d64de08342 P 1b4afce4018d4a93b66adbfd4b1f5e49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } }
I20260929 18:32:09.384925 31741 tablet_bootstrap.cc:492] T 13519206d3fc4377a65740d64de08342 P 50a58713495847f580c2a4e37fc4fd2b: No bootstrap required, opened a new log
I20260929 18:32:09.385028 31741 ts_tablet_manager.cc:1397] T 13519206d3fc4377a65740d64de08342 P 50a58713495847f580c2a4e37fc4fd2b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:32:09.385085 31740 ts_tablet_manager.cc:1428] T 13519206d3fc4377a65740d64de08342 P 1b4afce4018d4a93b66adbfd4b1f5e49: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:32:09.385118 31746 tablet_replica.cc:439] T 13519206d3fc4377a65740d64de08342 P 1b4afce4018d4a93b66adbfd4b1f5e49: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } } }
I20260929 18:32:09.385281 31746 tablet_replica.cc:442] T 13519206d3fc4377a65740d64de08342 P 1b4afce4018d4a93b66adbfd4b1f5e49: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:32:09.385562 31741 raft_consensus.cc:348] T 13519206d3fc4377a65740d64de08342 P 50a58713495847f580c2a4e37fc4fd2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } }
I20260929 18:32:09.385702 31741 raft_consensus.cc:374] T 13519206d3fc4377a65740d64de08342 P 50a58713495847f580c2a4e37fc4fd2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:09.385743 31741 raft_consensus.cc:729] T 13519206d3fc4377a65740d64de08342 P 50a58713495847f580c2a4e37fc4fd2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50a58713495847f580c2a4e37fc4fd2b, State: Initialized, Role: FOLLOWER
I20260929 18:32:09.385892 31741 consensus_queue.cc:260] T 13519206d3fc4377a65740d64de08342 P 50a58713495847f580c2a4e37fc4fd2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } }
I20260929 18:32:09.386193 31741 ts_tablet_manager.cc:1428] T 13519206d3fc4377a65740d64de08342 P 50a58713495847f580c2a4e37fc4fd2b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 18:32:09.386190 31748 tablet_replica.cc:439] T 13519206d3fc4377a65740d64de08342 P 50a58713495847f580c2a4e37fc4fd2b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } } }
I20260929 18:32:09.386785 31748 tablet_replica.cc:442] T 13519206d3fc4377a65740d64de08342 P 50a58713495847f580c2a4e37fc4fd2b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:32:09.386881 31742 tablet_bootstrap.cc:492] T 13519206d3fc4377a65740d64de08342 P e2b2ec819b274e2a94fdeca8a004a587: No bootstrap required, opened a new log
I20260929 18:32:09.386974 31742 ts_tablet_manager.cc:1397] T 13519206d3fc4377a65740d64de08342 P e2b2ec819b274e2a94fdeca8a004a587: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 18:32:09.387470 31742 raft_consensus.cc:348] T 13519206d3fc4377a65740d64de08342 P e2b2ec819b274e2a94fdeca8a004a587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } }
I20260929 18:32:09.387578 31742 raft_consensus.cc:374] T 13519206d3fc4377a65740d64de08342 P e2b2ec819b274e2a94fdeca8a004a587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:09.387599 31742 raft_consensus.cc:729] T 13519206d3fc4377a65740d64de08342 P e2b2ec819b274e2a94fdeca8a004a587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2b2ec819b274e2a94fdeca8a004a587, State: Initialized, Role: FOLLOWER
I20260929 18:32:09.387750 31742 consensus_queue.cc:260] T 13519206d3fc4377a65740d64de08342 P e2b2ec819b274e2a94fdeca8a004a587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } }
I20260929 18:32:09.388051 31742 ts_tablet_manager.cc:1428] T 13519206d3fc4377a65740d64de08342 P e2b2ec819b274e2a94fdeca8a004a587: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 18:32:09.388047 31747 tablet_replica.cc:439] T 13519206d3fc4377a65740d64de08342 P e2b2ec819b274e2a94fdeca8a004a587: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } } }
I20260929 18:32:09.388214 31747 tablet_replica.cc:442] T 13519206d3fc4377a65740d64de08342 P e2b2ec819b274e2a94fdeca8a004a587: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:32:09.472668 31748 raft_consensus.cc:482] T 13519206d3fc4377a65740d64de08342 P 50a58713495847f580c2a4e37fc4fd2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:32:09.472838 31748 raft_consensus.cc:504] T 13519206d3fc4377a65740d64de08342 P 50a58713495847f580c2a4e37fc4fd2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } }
I20260929 18:32:09.473254 31748 leader_election.cc:288] T 13519206d3fc4377a65740d64de08342 P 50a58713495847f580c2a4e37fc4fd2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b4afce4018d4a93b66adbfd4b1f5e49 (127.28.207.65:46387), e2b2ec819b274e2a94fdeca8a004a587 (127.28.207.66:41597)
I20260929 18:32:09.474042 31391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13519206d3fc4377a65740d64de08342" candidate_uuid: "50a58713495847f580c2a4e37fc4fd2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" is_pre_election: true
I20260929 18:32:09.474051 31524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13519206d3fc4377a65740d64de08342" candidate_uuid: "50a58713495847f580c2a4e37fc4fd2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2b2ec819b274e2a94fdeca8a004a587" is_pre_election: true
I20260929 18:32:09.474288 31391 raft_consensus.cc:2454] T 13519206d3fc4377a65740d64de08342 P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50a58713495847f580c2a4e37fc4fd2b in term 0.
I20260929 18:32:09.474288 31524 raft_consensus.cc:2454] T 13519206d3fc4377a65740d64de08342 P e2b2ec819b274e2a94fdeca8a004a587 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50a58713495847f580c2a4e37fc4fd2b in term 0.
I20260929 18:32:09.474766 31604 leader_election.cc:302] T 13519206d3fc4377a65740d64de08342 P 50a58713495847f580c2a4e37fc4fd2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b4afce4018d4a93b66adbfd4b1f5e49, 50a58713495847f580c2a4e37fc4fd2b; no voters:
I20260929 18:32:09.475193 31748 raft_consensus.cc:2781] T 13519206d3fc4377a65740d64de08342 P 50a58713495847f580c2a4e37fc4fd2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:32:09.475301 31748 raft_consensus.cc:482] T 13519206d3fc4377a65740d64de08342 P 50a58713495847f580c2a4e37fc4fd2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:32:09.475364 31748 raft_consensus.cc:3037] T 13519206d3fc4377a65740d64de08342 P 50a58713495847f580c2a4e37fc4fd2b [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:09.476555 31748 raft_consensus.cc:504] T 13519206d3fc4377a65740d64de08342 P 50a58713495847f580c2a4e37fc4fd2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } }
I20260929 18:32:09.477056 31748 leader_election.cc:288] T 13519206d3fc4377a65740d64de08342 P 50a58713495847f580c2a4e37fc4fd2b [CANDIDATE]: Term 1 election: Requested vote from peers 1b4afce4018d4a93b66adbfd4b1f5e49 (127.28.207.65:46387), e2b2ec819b274e2a94fdeca8a004a587 (127.28.207.66:41597)
I20260929 18:32:09.477581 31524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13519206d3fc4377a65740d64de08342" candidate_uuid: "50a58713495847f580c2a4e37fc4fd2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2b2ec819b274e2a94fdeca8a004a587"
I20260929 18:32:09.477603 31391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13519206d3fc4377a65740d64de08342" candidate_uuid: "50a58713495847f580c2a4e37fc4fd2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49"
I20260929 18:32:09.477751 31524 raft_consensus.cc:3037] T 13519206d3fc4377a65740d64de08342 P e2b2ec819b274e2a94fdeca8a004a587 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:09.477818 31391 raft_consensus.cc:3037] T 13519206d3fc4377a65740d64de08342 P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:09.479121 31391 raft_consensus.cc:2454] T 13519206d3fc4377a65740d64de08342 P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50a58713495847f580c2a4e37fc4fd2b in term 1.
I20260929 18:32:09.479319 31524 raft_consensus.cc:2454] T 13519206d3fc4377a65740d64de08342 P e2b2ec819b274e2a94fdeca8a004a587 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50a58713495847f580c2a4e37fc4fd2b in term 1.
I20260929 18:32:09.479490 31604 leader_election.cc:302] T 13519206d3fc4377a65740d64de08342 P 50a58713495847f580c2a4e37fc4fd2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b4afce4018d4a93b66adbfd4b1f5e49, 50a58713495847f580c2a4e37fc4fd2b; no voters:
I20260929 18:32:09.479707 31748 raft_consensus.cc:2781] T 13519206d3fc4377a65740d64de08342 P 50a58713495847f580c2a4e37fc4fd2b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:32:09.479849 31748 raft_consensus.cc:686] T 13519206d3fc4377a65740d64de08342 P 50a58713495847f580c2a4e37fc4fd2b [term 1 LEADER]: Becoming Leader. State: Replica: 50a58713495847f580c2a4e37fc4fd2b, State: Running, Role: LEADER
I20260929 18:32:09.480050 31748 consensus_queue.cc:237] T 13519206d3fc4377a65740d64de08342 P 50a58713495847f580c2a4e37fc4fd2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } }
I20260929 18:32:09.481555 31750 tablet_replica.cc:439] T 13519206d3fc4377a65740d64de08342 P 50a58713495847f580c2a4e37fc4fd2b: TxnStatusTablet state changed. Reason: New leader 50a58713495847f580c2a4e37fc4fd2b. Latest consensus state: current_term: 1 leader_uuid: "50a58713495847f580c2a4e37fc4fd2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } } }
I20260929 18:32:09.481719 31750 tablet_replica.cc:442] T 13519206d3fc4377a65740d64de08342 P 50a58713495847f580c2a4e37fc4fd2b: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:32:09.482096 31762 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 18:32:09.482342 31762 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 18:32:09.482827 31233 catalog_manager.cc:5495] T 13519206d3fc4377a65740d64de08342 P 50a58713495847f580c2a4e37fc4fd2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 50a58713495847f580c2a4e37fc4fd2b (127.28.207.67). New cstate: current_term: 1 leader_uuid: "50a58713495847f580c2a4e37fc4fd2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } health_report { overall_health: HEALTHY } } }
I20260929 18:32:09.515231 31738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:09.510819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52948 (local address 127.28.207.67:34053)
0929 18:32:09.511041 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:09.511047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:09.511065 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:32:09.511233 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:09.511236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:09.511300 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:09.511414 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:32:09.511936 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.512573 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.513516 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.513748 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.513866 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:09.514176 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:32:09.514754 (+ 578us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:32:09.514920 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:09.515000 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 18:32:09.515063 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260929 18:32:09.515223 31757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:09.510649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:34053 (local address 127.0.0.1:52948)
0929 18:32:09.510843 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:09.510889 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:09.511037 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:09.511430 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:09.511433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:09.511441 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:32:09.511710 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:09.511730 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.512724 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.512730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:09.513385 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:09.513394 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.513869 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.513873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:09.513963 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:09.514018 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:32:09.514891 (+ 873us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:32:09.514909 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 18:32:09.514987 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 18:32:09.515080 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 18:32:09.523766 31391 raft_consensus.cc:1261] T 13519206d3fc4377a65740d64de08342 P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 1 FOLLOWER]: Refusing update from remote peer 50a58713495847f580c2a4e37fc4fd2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:32:09.523775 31524 raft_consensus.cc:1261] T 13519206d3fc4377a65740d64de08342 P e2b2ec819b274e2a94fdeca8a004a587 [term 1 FOLLOWER]: Refusing update from remote peer 50a58713495847f580c2a4e37fc4fd2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:32:09.524658 31748 consensus_queue.cc:1035] T 13519206d3fc4377a65740d64de08342 P 50a58713495847f580c2a4e37fc4fd2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:32:09.524891 31750 consensus_queue.cc:1035] T 13519206d3fc4377a65740d64de08342 P 50a58713495847f580c2a4e37fc4fd2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:32:09.529649 31747 tablet_replica.cc:439] T 13519206d3fc4377a65740d64de08342 P e2b2ec819b274e2a94fdeca8a004a587: TxnStatusTablet state changed. Reason: New leader 50a58713495847f580c2a4e37fc4fd2b. Latest consensus state: current_term: 1 leader_uuid: "50a58713495847f580c2a4e37fc4fd2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } } }
I20260929 18:32:09.529826 31747 tablet_replica.cc:442] T 13519206d3fc4377a65740d64de08342 P e2b2ec819b274e2a94fdeca8a004a587: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:32:09.529847 31746 tablet_replica.cc:439] T 13519206d3fc4377a65740d64de08342 P 1b4afce4018d4a93b66adbfd4b1f5e49: TxnStatusTablet state changed. Reason: New leader 50a58713495847f580c2a4e37fc4fd2b. Latest consensus state: current_term: 1 leader_uuid: "50a58713495847f580c2a4e37fc4fd2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } } }
I20260929 18:32:09.530023 31746 tablet_replica.cc:442] T 13519206d3fc4377a65740d64de08342 P 1b4afce4018d4a93b66adbfd4b1f5e49: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:32:09.535125 31748 tablet_replica.cc:439] T 13519206d3fc4377a65740d64de08342 P 50a58713495847f580c2a4e37fc4fd2b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "50a58713495847f580c2a4e37fc4fd2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } } }
I20260929 18:32:09.535315 31748 tablet_replica.cc:442] T 13519206d3fc4377a65740d64de08342 P 50a58713495847f580c2a4e37fc4fd2b: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:32:09.535758 31747 tablet_replica.cc:439] T 13519206d3fc4377a65740d64de08342 P e2b2ec819b274e2a94fdeca8a004a587: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "50a58713495847f580c2a4e37fc4fd2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } } }
I20260929 18:32:09.535916 31747 tablet_replica.cc:442] T 13519206d3fc4377a65740d64de08342 P e2b2ec819b274e2a94fdeca8a004a587: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:32:09.535949 31765 mvcc.cc:205] Tried to move back new op lower bound from 7334734764119031808 to 7334734763953401856. Current Snapshot: MvccSnapshot[applied={T|T < 7334734764119031808}]
I20260929 18:32:09.537324 31750 tablet_replica.cc:439] T 13519206d3fc4377a65740d64de08342 P 50a58713495847f580c2a4e37fc4fd2b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "50a58713495847f580c2a4e37fc4fd2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } } }
I20260929 18:32:09.537456 31750 tablet_replica.cc:442] T 13519206d3fc4377a65740d64de08342 P 50a58713495847f580c2a4e37fc4fd2b: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:32:09.540058 31746 tablet_replica.cc:439] T 13519206d3fc4377a65740d64de08342 P 1b4afce4018d4a93b66adbfd4b1f5e49: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "50a58713495847f580c2a4e37fc4fd2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } } }
I20260929 18:32:09.540222 31746 tablet_replica.cc:442] T 13519206d3fc4377a65740d64de08342 P 1b4afce4018d4a93b66adbfd4b1f5e49: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:32:09.544443 31767 mvcc.cc:205] Tried to move back new op lower bound from 7334734764119031808 to 7334734763953401856. Current Snapshot: MvccSnapshot[applied={T|T < 7334734764119031808}]
I20260929 18:32:09.547542 31738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:09.543582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52954 (local address 127.28.207.67:34053)
0929 18:32:09.543718 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:09.543723 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:09.543786 (+ 63us) server_negotiation.cc:407] Connection header received
0929 18:32:09.543969 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:09.543973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:09.544036 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:09.544129 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:32:09.544411 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.545214 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.546318 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.546612 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.546716 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:09.547054 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:32:09.547146 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:32:09.547228 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:09.547320 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 18:32:09.547379 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:32:09.562631 31738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:09.557906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:36647 (local address 127.28.207.67:34053)
0929 18:32:09.558328 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:09.558332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:09.558348 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:32:09.558394 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:09.558397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:09.558447 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:09.558556 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:32:09.559131 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.559774 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.561172 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.561413 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.561532 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:09.561971 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:32:09.562064 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:32:09.562142 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:09.562315 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 18:32:09.562396 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319}
I20260929 18:32:09.562647 31723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:09.557768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:34053 (local address 127.28.207.67:36647)
0929 18:32:09.557935 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:09.557953 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:09.558063 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:09.558604 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:09.558607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:09.558616 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:32:09.558770 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:09.558777 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.559918 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.559922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:09.560947 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:09.560959 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.561569 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.561574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:09.561682 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:09.561747 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:32:09.562182 (+ 435us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:32:09.562190 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 18:32:09.562280 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 18:32:09.562367 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 18:32:09.571508 31391 raft_consensus.cc:1261] T b21c5b1e8621467ba7fb32400dabf70e P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 1 FOLLOWER]: Refusing update from remote peer 50a58713495847f580c2a4e37fc4fd2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:32:09.571578 31524 raft_consensus.cc:1261] T b21c5b1e8621467ba7fb32400dabf70e P e2b2ec819b274e2a94fdeca8a004a587 [term 1 FOLLOWER]: Refusing update from remote peer 50a58713495847f580c2a4e37fc4fd2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:32:09.572093 31748 consensus_queue.cc:1035] T b21c5b1e8621467ba7fb32400dabf70e P 50a58713495847f580c2a4e37fc4fd2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:32:09.572387 31750 consensus_queue.cc:1035] T b21c5b1e8621467ba7fb32400dabf70e P 50a58713495847f580c2a4e37fc4fd2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:32:09.628882 31233 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:32:09.629179 31233 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:32:09.632567 31233 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d27777167194a44a0b5510f5b8d6c99: Sending DeleteTablet for 3 replicas of tablet b21c5b1e8621467ba7fb32400dabf70e
I20260929 18:32:09.633754 31371 tablet_service.cc:1514] Processing DeleteTablet for tablet b21c5b1e8621467ba7fb32400dabf70e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:09 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:43920
I20260929 18:32:09.633961 31511 tablet_service.cc:1514] Processing DeleteTablet for tablet b21c5b1e8621467ba7fb32400dabf70e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:09 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:41982
I20260929 18:32:09.634687 31781 tablet_replica.cc:330] T b21c5b1e8621467ba7fb32400dabf70e P 1b4afce4018d4a93b66adbfd4b1f5e49: stopping tablet replica
I20260929 18:32:09.634760 31650 tablet_service.cc:1514] Processing DeleteTablet for tablet b21c5b1e8621467ba7fb32400dabf70e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:09 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:52934
I20260929 18:32:09.635010 31233 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 18:32:09.635281 31781 raft_consensus.cc:2229] T b21c5b1e8621467ba7fb32400dabf70e P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260929 18:32:09.635608 31233 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:32:09.635630 31781 raft_consensus.cc:2258] T b21c5b1e8621467ba7fb32400dabf70e P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:09.635666 31782 tablet_replica.cc:330] T b21c5b1e8621467ba7fb32400dabf70e P e2b2ec819b274e2a94fdeca8a004a587: stopping tablet replica
I20260929 18:32:09.636237 31782 raft_consensus.cc:2229] T b21c5b1e8621467ba7fb32400dabf70e P e2b2ec819b274e2a94fdeca8a004a587 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:32:09.636560 31782 raft_consensus.cc:2258] T b21c5b1e8621467ba7fb32400dabf70e P e2b2ec819b274e2a94fdeca8a004a587 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:09.636809 31783 tablet_replica.cc:330] T b21c5b1e8621467ba7fb32400dabf70e P 50a58713495847f580c2a4e37fc4fd2b: stopping tablet replica
I20260929 18:32:09.637261 31783 raft_consensus.cc:2229] T b21c5b1e8621467ba7fb32400dabf70e P 50a58713495847f580c2a4e37fc4fd2b [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:32:09.637827 31783 raft_consensus.cc:2258] T b21c5b1e8621467ba7fb32400dabf70e P 50a58713495847f580c2a4e37fc4fd2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:09.638345 31782 ts_tablet_manager.cc:1905] T b21c5b1e8621467ba7fb32400dabf70e P e2b2ec819b274e2a94fdeca8a004a587: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:32:09.639729 31783 ts_tablet_manager.cc:1905] T b21c5b1e8621467ba7fb32400dabf70e P 50a58713495847f580c2a4e37fc4fd2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:32:09.642196 31782 ts_tablet_manager.cc:1918] T b21c5b1e8621467ba7fb32400dabf70e P e2b2ec819b274e2a94fdeca8a004a587: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:32:09.642463 31782 log.cc:1196] T b21c5b1e8621467ba7fb32400dabf70e P e2b2ec819b274e2a94fdeca8a004a587: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-1/wal/wals/b21c5b1e8621467ba7fb32400dabf70e
I20260929 18:32:09.643003 31782 ts_tablet_manager.cc:1939] T b21c5b1e8621467ba7fb32400dabf70e P e2b2ec819b274e2a94fdeca8a004a587: Deleting consensus metadata
I20260929 18:32:09.643846 31783 ts_tablet_manager.cc:1918] T b21c5b1e8621467ba7fb32400dabf70e P 50a58713495847f580c2a4e37fc4fd2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:32:09.644044 31783 log.cc:1196] T b21c5b1e8621467ba7fb32400dabf70e P 50a58713495847f580c2a4e37fc4fd2b: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-2/wal/wals/b21c5b1e8621467ba7fb32400dabf70e
I20260929 18:32:09.644521 31783 ts_tablet_manager.cc:1939] T b21c5b1e8621467ba7fb32400dabf70e P 50a58713495847f580c2a4e37fc4fd2b: Deleting consensus metadata
I20260929 18:32:09.644640 31221 catalog_manager.cc:4841] TS e2b2ec819b274e2a94fdeca8a004a587 (127.28.207.66:41597): tablet b21c5b1e8621467ba7fb32400dabf70e (table test-table [id=e4932755fb2140dab385d6ad43f4a257]) successfully deleted
I20260929 18:32:09.645464 31511 tablet_service.cc:1467] Processing CreateTablet for tablet 40f9a7e1bf4c4e08a5f06f4b7b99ba0c (DEFAULT_TABLE table=test-table [id=b4911621830d4894b7431fc13470bf4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:09.645754 31511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40f9a7e1bf4c4e08a5f06f4b7b99ba0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:09.646195 31371 tablet_service.cc:1467] Processing CreateTablet for tablet 40f9a7e1bf4c4e08a5f06f4b7b99ba0c (DEFAULT_TABLE table=test-table [id=b4911621830d4894b7431fc13470bf4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:09.646312 31220 catalog_manager.cc:4841] TS 50a58713495847f580c2a4e37fc4fd2b (127.28.207.67:34053): tablet b21c5b1e8621467ba7fb32400dabf70e (table test-table [id=e4932755fb2140dab385d6ad43f4a257]) successfully deleted
I20260929 18:32:09.646613 31371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40f9a7e1bf4c4e08a5f06f4b7b99ba0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:09.647153 31650 tablet_service.cc:1467] Processing CreateTablet for tablet 40f9a7e1bf4c4e08a5f06f4b7b99ba0c (DEFAULT_TABLE table=test-table [id=b4911621830d4894b7431fc13470bf4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:09.647454 31650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40f9a7e1bf4c4e08a5f06f4b7b99ba0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:09.649013 31742 tablet_bootstrap.cc:492] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P e2b2ec819b274e2a94fdeca8a004a587: Bootstrap starting.
I20260929 18:32:09.650156 31742 tablet_bootstrap.cc:654] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P e2b2ec819b274e2a94fdeca8a004a587: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:09.650452 31741 tablet_bootstrap.cc:492] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 50a58713495847f580c2a4e37fc4fd2b: Bootstrap starting.
I20260929 18:32:09.650913 31740 tablet_bootstrap.cc:492] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 1b4afce4018d4a93b66adbfd4b1f5e49: Bootstrap starting.
I20260929 18:32:09.651618 31741 tablet_bootstrap.cc:654] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 50a58713495847f580c2a4e37fc4fd2b: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:09.652064 31740 tablet_bootstrap.cc:654] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 1b4afce4018d4a93b66adbfd4b1f5e49: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:09.652283 31742 tablet_bootstrap.cc:492] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P e2b2ec819b274e2a94fdeca8a004a587: No bootstrap required, opened a new log
I20260929 18:32:09.652370 31742 ts_tablet_manager.cc:1397] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P e2b2ec819b274e2a94fdeca8a004a587: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:32:09.652850 31742 raft_consensus.cc:348] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P e2b2ec819b274e2a94fdeca8a004a587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } }
I20260929 18:32:09.652973 31742 raft_consensus.cc:374] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P e2b2ec819b274e2a94fdeca8a004a587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:09.653018 31742 raft_consensus.cc:729] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P e2b2ec819b274e2a94fdeca8a004a587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2b2ec819b274e2a94fdeca8a004a587, State: Initialized, Role: FOLLOWER
I20260929 18:32:09.653156 31742 consensus_queue.cc:260] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P e2b2ec819b274e2a94fdeca8a004a587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } }
I20260929 18:32:09.653427 31742 ts_tablet_manager.cc:1428] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P e2b2ec819b274e2a94fdeca8a004a587: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:32:09.654035 31781 ts_tablet_manager.cc:1905] T b21c5b1e8621467ba7fb32400dabf70e P 1b4afce4018d4a93b66adbfd4b1f5e49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:32:09.655305 31741 tablet_bootstrap.cc:492] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 50a58713495847f580c2a4e37fc4fd2b: No bootstrap required, opened a new log
I20260929 18:32:09.655409 31741 ts_tablet_manager.cc:1397] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 50a58713495847f580c2a4e37fc4fd2b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260929 18:32:09.655944 31741 raft_consensus.cc:348] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 50a58713495847f580c2a4e37fc4fd2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } }
I20260929 18:32:09.656090 31741 raft_consensus.cc:374] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 50a58713495847f580c2a4e37fc4fd2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:09.656131 31741 raft_consensus.cc:729] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 50a58713495847f580c2a4e37fc4fd2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50a58713495847f580c2a4e37fc4fd2b, State: Initialized, Role: FOLLOWER
I20260929 18:32:09.656921 31740 tablet_bootstrap.cc:492] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 1b4afce4018d4a93b66adbfd4b1f5e49: No bootstrap required, opened a new log
I20260929 18:32:09.657063 31740 ts_tablet_manager.cc:1397] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 1b4afce4018d4a93b66adbfd4b1f5e49: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260929 18:32:09.657282 31741 consensus_queue.cc:260] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 50a58713495847f580c2a4e37fc4fd2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } }
I20260929 18:32:09.657626 31740 raft_consensus.cc:348] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } }
I20260929 18:32:09.657766 31740 raft_consensus.cc:374] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:09.657800 31740 raft_consensus.cc:729] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b4afce4018d4a93b66adbfd4b1f5e49, State: Initialized, Role: FOLLOWER
I20260929 18:32:09.657965 31781 ts_tablet_manager.cc:1918] T b21c5b1e8621467ba7fb32400dabf70e P 1b4afce4018d4a93b66adbfd4b1f5e49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:32:09.658231 31781 log.cc:1196] T b21c5b1e8621467ba7fb32400dabf70e P 1b4afce4018d4a93b66adbfd4b1f5e49: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-0/wal/wals/b21c5b1e8621467ba7fb32400dabf70e
I20260929 18:32:09.657953 31740 consensus_queue.cc:260] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 1b4afce4018d4a93b66adbfd4b1f5e49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } }
I20260929 18:32:09.658591 31740 ts_tablet_manager.cc:1428] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 1b4afce4018d4a93b66adbfd4b1f5e49: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 18:32:09.659534 31781 ts_tablet_manager.cc:1939] T b21c5b1e8621467ba7fb32400dabf70e P 1b4afce4018d4a93b66adbfd4b1f5e49: Deleting consensus metadata
I20260929 18:32:09.659981 31741 ts_tablet_manager.cc:1428] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 50a58713495847f580c2a4e37fc4fd2b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260929 18:32:09.661188 31220 catalog_manager.cc:4841] TS 1b4afce4018d4a93b66adbfd4b1f5e49 (127.28.207.65:46387): tablet b21c5b1e8621467ba7fb32400dabf70e (table test-table [id=e4932755fb2140dab385d6ad43f4a257]) successfully deleted
I20260929 18:32:09.818012 31746 raft_consensus.cc:482] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:32:09.818210 31746 raft_consensus.cc:504] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } }
I20260929 18:32:09.819743 31746 leader_election.cc:288] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 1b4afce4018d4a93b66adbfd4b1f5e49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2b2ec819b274e2a94fdeca8a004a587 (127.28.207.66:41597), 50a58713495847f580c2a4e37fc4fd2b (127.28.207.67:34053)
I20260929 18:32:09.825670 31738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:09.820554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:45617 (local address 127.28.207.67:34053)
0929 18:32:09.820683 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:09.820688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:09.820713 (+ 25us) server_negotiation.cc:407] Connection header received
0929 18:32:09.820878 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:09.820881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:09.820937 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:09.821027 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:09.821375 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.822073 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.824145 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.824928 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.825108 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:09.825181 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:09.825421 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 18:32:09.825529 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:32:09.825685 31788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:09.820145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:34053 (local address 127.28.207.65:45617)
0929 18:32:09.820644 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:09.820659 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:09.820733 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:09.821092 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:09.821095 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:09.821107 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:09.821223 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:09.821229 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.822333 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.822336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:09.823977 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:09.823990 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.825122 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.825125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:09.825241 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:09.825270 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 18:32:09.825394 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 18:32:09.825491 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":185,"thread_start_us":88,"threads_started":1}
I20260929 18:32:09.826484 31671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40f9a7e1bf4c4e08a5f06f4b7b99ba0c" candidate_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50a58713495847f580c2a4e37fc4fd2b" is_pre_election: true
I20260929 18:32:09.826829 31671 raft_consensus.cc:2454] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 50a58713495847f580c2a4e37fc4fd2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b4afce4018d4a93b66adbfd4b1f5e49 in term 0.
I20260929 18:32:09.826961 31787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:09.820020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:41597 (local address 127.28.207.65:38377)
0929 18:32:09.820444 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:09.820463 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:09.820590 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:09.820961 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:09.820964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:09.820980 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:09.821109 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:09.821116 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.822277 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.822280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:09.823977 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:09.823990 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.826458 (+ 2468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.826462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:09.826569 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:09.826593 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 18:32:09.826683 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 18:32:09.826765 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":103,"thread_start_us":113,"threads_started":1}
I20260929 18:32:09.827160 31789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:09.820196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:38377 (local address 127.28.207.66:41597)
0929 18:32:09.820496 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:09.820504 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:09.820543 (+ 39us) server_negotiation.cc:407] Connection header received
0929 18:32:09.820752 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:09.820758 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:09.820825 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:09.820939 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:09.821258 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.822109 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.825249 (+ 3140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.826294 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.826422 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:09.826474 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:09.826882 (+ 408us) server_negotiation.cc:299] Negotiation successful
0929 18:32:09.826983 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":71,"threads_started":1}
I20260929 18:32:09.827412 31324 leader_election.cc:302] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 1b4afce4018d4a93b66adbfd4b1f5e49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b4afce4018d4a93b66adbfd4b1f5e49, 50a58713495847f580c2a4e37fc4fd2b; no voters:
I20260929 18:32:09.827538 31531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40f9a7e1bf4c4e08a5f06f4b7b99ba0c" candidate_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2b2ec819b274e2a94fdeca8a004a587" is_pre_election: true
I20260929 18:32:09.827688 31531 raft_consensus.cc:2454] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P e2b2ec819b274e2a94fdeca8a004a587 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b4afce4018d4a93b66adbfd4b1f5e49 in term 0.
I20260929 18:32:09.827755 31746 raft_consensus.cc:2781] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:32:09.827816 31746 raft_consensus.cc:482] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:32:09.827853 31746 raft_consensus.cc:3037] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:09.829123 31746 raft_consensus.cc:504] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } }
I20260929 18:32:09.829572 31746 leader_election.cc:288] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 1b4afce4018d4a93b66adbfd4b1f5e49 [CANDIDATE]: Term 1 election: Requested vote from peers e2b2ec819b274e2a94fdeca8a004a587 (127.28.207.66:41597), 50a58713495847f580c2a4e37fc4fd2b (127.28.207.67:34053)
I20260929 18:32:09.830003 31531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40f9a7e1bf4c4e08a5f06f4b7b99ba0c" candidate_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2b2ec819b274e2a94fdeca8a004a587"
I20260929 18:32:09.830164 31531 raft_consensus.cc:3037] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P e2b2ec819b274e2a94fdeca8a004a587 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:09.830369 31671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40f9a7e1bf4c4e08a5f06f4b7b99ba0c" candidate_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50a58713495847f580c2a4e37fc4fd2b"
I20260929 18:32:09.830535 31671 raft_consensus.cc:3037] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 50a58713495847f580c2a4e37fc4fd2b [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:09.831425 31531 raft_consensus.cc:2454] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P e2b2ec819b274e2a94fdeca8a004a587 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b4afce4018d4a93b66adbfd4b1f5e49 in term 1.
I20260929 18:32:09.831848 31325 leader_election.cc:302] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 1b4afce4018d4a93b66adbfd4b1f5e49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b4afce4018d4a93b66adbfd4b1f5e49, e2b2ec819b274e2a94fdeca8a004a587; no voters:
I20260929 18:32:09.831939 31671 raft_consensus.cc:2454] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 50a58713495847f580c2a4e37fc4fd2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b4afce4018d4a93b66adbfd4b1f5e49 in term 1.
I20260929 18:32:09.832062 31746 raft_consensus.cc:2781] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:32:09.832383 31746 raft_consensus.cc:686] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 1 LEADER]: Becoming Leader. State: Replica: 1b4afce4018d4a93b66adbfd4b1f5e49, State: Running, Role: LEADER
I20260929 18:32:09.832728 31746 consensus_queue.cc:237] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 1b4afce4018d4a93b66adbfd4b1f5e49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } }
I20260929 18:32:09.836192 31234 catalog_manager.cc:5495] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 1b4afce4018d4a93b66adbfd4b1f5e49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b4afce4018d4a93b66adbfd4b1f5e49 (127.28.207.65). New cstate: current_term: 1 leader_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } health_report { overall_health: UNKNOWN } } }
I20260929 18:32:09.863497 31792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:09.857991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43924 (local address 127.28.207.65:46387)
0929 18:32:09.858292 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:09.858298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:09.858310 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:32:09.858453 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:09.858463 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:09.858573 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:09.858741 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:32:09.859128 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.860491 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.861462 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.861718 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.861818 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:09.862174 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:32:09.862786 (+ 612us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:32:09.863003 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:09.863212 (+ 209us) server_negotiation.cc:299] Negotiation successful
0929 18:32:09.863308 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":145,"threads_started":1}
I20260929 18:32:09.877693 31738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:09.873714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:35639 (local address 127.28.207.67:34053)
0929 18:32:09.873847 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:09.873851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:09.874056 (+ 205us) server_negotiation.cc:407] Connection header received
0929 18:32:09.874254 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:09.874257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:09.874307 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:09.874403 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:32:09.874765 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.875410 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.876393 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.876626 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.876731 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:09.877156 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:32:09.877235 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:32:09.877312 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:09.877517 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 18:32:09.877581 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:32:09.877727 31795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:09.873557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:34053 (local address 127.28.207.65:35639)
0929 18:32:09.873924 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:09.873938 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:09.874106 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:09.874438 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:09.874441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:09.874448 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:32:09.874604 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:09.874609 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.875541 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.875545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:09.876254 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:09.876262 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.876799 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.876804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:09.876918 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:09.876976 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:32:09.877365 (+ 389us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:32:09.877382 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 18:32:09.877478 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 18:32:09.877572 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":149,"threads_started":1}
I20260929 18:32:09.885885 31531 raft_consensus.cc:1261] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P e2b2ec819b274e2a94fdeca8a004a587 [term 1 FOLLOWER]: Refusing update from remote peer 1b4afce4018d4a93b66adbfd4b1f5e49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:32:09.885921 31671 raft_consensus.cc:1261] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 50a58713495847f580c2a4e37fc4fd2b [term 1 FOLLOWER]: Refusing update from remote peer 1b4afce4018d4a93b66adbfd4b1f5e49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:32:09.886456 31790 consensus_queue.cc:1035] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 1b4afce4018d4a93b66adbfd4b1f5e49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:32:09.886651 31746 consensus_queue.cc:1035] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 1b4afce4018d4a93b66adbfd4b1f5e49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:32:09.909504 31792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:09.905364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:39907 (local address 127.28.207.65:46387)
0929 18:32:09.905510 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:09.905514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:09.905589 (+ 75us) server_negotiation.cc:407] Connection header received
0929 18:32:09.905875 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:09.905878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:09.905996 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:09.906092 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:32:09.906592 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.907340 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.908374 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.908623 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.908738 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:09.908987 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:32:09.909079 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:32:09.909158 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:09.909246 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 18:32:09.909303 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:32:09.909504 31723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:09.905278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:46387 (local address 127.28.207.67:39907)
0929 18:32:09.905492 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:09.905533 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:09.905633 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:09.906162 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:09.906166 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:09.906174 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:32:09.906305 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:09.906316 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.907453 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.907459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:09.908229 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:09.908241 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:09.908737 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:09.908741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:09.908825 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:09.908866 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:32:09.909162 (+ 296us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:32:09.909171 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 18:32:09.909239 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 18:32:09.909329 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127}
W20260929 18:32:09.910724 31767 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 18:32:11.488967 31234 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:32:11.489171 31234 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:32:11.493019 31234 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d27777167194a44a0b5510f5b8d6c99: Sending DeleteTablet for 3 replicas of tablet 40f9a7e1bf4c4e08a5f06f4b7b99ba0c
I20260929 18:32:11.494385 31511 tablet_service.cc:1514] Processing DeleteTablet for tablet 40f9a7e1bf4c4e08a5f06f4b7b99ba0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:11 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:41982
I20260929 18:32:11.494416 31371 tablet_service.cc:1514] Processing DeleteTablet for tablet 40f9a7e1bf4c4e08a5f06f4b7b99ba0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:11 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:43920
I20260929 18:32:11.494786 31650 tablet_service.cc:1514] Processing DeleteTablet for tablet 40f9a7e1bf4c4e08a5f06f4b7b99ba0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:11 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:52934
I20260929 18:32:11.494864 31823 tablet_replica.cc:330] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 1b4afce4018d4a93b66adbfd4b1f5e49: stopping tablet replica
I20260929 18:32:11.494915 31822 tablet_replica.cc:330] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P e2b2ec819b274e2a94fdeca8a004a587: stopping tablet replica
I20260929 18:32:11.495051 31823 raft_consensus.cc:2229] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:32:11.495050 31822 raft_consensus.cc:2229] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P e2b2ec819b274e2a94fdeca8a004a587 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:32:11.495198 31822 raft_consensus.cc:2258] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P e2b2ec819b274e2a94fdeca8a004a587 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:11.495514 31823 raft_consensus.cc:2258] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:11.496025 31824 tablet_replica.cc:330] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 50a58713495847f580c2a4e37fc4fd2b: stopping tablet replica
I20260929 18:32:11.496140 31822 ts_tablet_manager.cc:1905] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P e2b2ec819b274e2a94fdeca8a004a587: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:32:11.496233 31823 ts_tablet_manager.cc:1905] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 1b4afce4018d4a93b66adbfd4b1f5e49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:32:11.496299 31824 raft_consensus.cc:2229] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 50a58713495847f580c2a4e37fc4fd2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:32:11.496532 31824 raft_consensus.cc:2258] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 50a58713495847f580c2a4e37fc4fd2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:11.499859 31822 ts_tablet_manager.cc:1918] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P e2b2ec819b274e2a94fdeca8a004a587: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 18:32:11.499943 31823 ts_tablet_manager.cc:1918] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 1b4afce4018d4a93b66adbfd4b1f5e49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 18:32:11.500041 31822 log.cc:1196] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P e2b2ec819b274e2a94fdeca8a004a587: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-1/wal/wals/40f9a7e1bf4c4e08a5f06f4b7b99ba0c
I20260929 18:32:11.500080 31823 log.cc:1196] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 1b4afce4018d4a93b66adbfd4b1f5e49: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-0/wal/wals/40f9a7e1bf4c4e08a5f06f4b7b99ba0c
I20260929 18:32:11.500450 31823 ts_tablet_manager.cc:1939] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 1b4afce4018d4a93b66adbfd4b1f5e49: Deleting consensus metadata
I20260929 18:32:11.500450 31822 ts_tablet_manager.cc:1939] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P e2b2ec819b274e2a94fdeca8a004a587: Deleting consensus metadata
I20260929 18:32:11.501596 31221 catalog_manager.cc:4841] TS e2b2ec819b274e2a94fdeca8a004a587 (127.28.207.66:41597): tablet 40f9a7e1bf4c4e08a5f06f4b7b99ba0c (table test-table [id=b4911621830d4894b7431fc13470bf4a]) successfully deleted
I20260929 18:32:11.501652 31220 catalog_manager.cc:4841] TS 1b4afce4018d4a93b66adbfd4b1f5e49 (127.28.207.65:46387): tablet 40f9a7e1bf4c4e08a5f06f4b7b99ba0c (table test-table [id=b4911621830d4894b7431fc13470bf4a]) successfully deleted
I20260929 18:32:11.502380 31824 ts_tablet_manager.cc:1905] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 50a58713495847f580c2a4e37fc4fd2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:32:11.505623 31824 ts_tablet_manager.cc:1918] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 50a58713495847f580c2a4e37fc4fd2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 18:32:11.505802 31824 log.cc:1196] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 50a58713495847f580c2a4e37fc4fd2b: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-2/wal/wals/40f9a7e1bf4c4e08a5f06f4b7b99ba0c
I20260929 18:32:11.506120 31824 ts_tablet_manager.cc:1939] T 40f9a7e1bf4c4e08a5f06f4b7b99ba0c P 50a58713495847f580c2a4e37fc4fd2b: Deleting consensus metadata
I20260929 18:32:11.508657 31220 catalog_manager.cc:4841] TS 50a58713495847f580c2a4e37fc4fd2b (127.28.207.67:34053): tablet 40f9a7e1bf4c4e08a5f06f4b7b99ba0c (table test-table [id=b4911621830d4894b7431fc13470bf4a]) successfully deleted
Sep 29 18:32:11 dist-test-slave-qcpv krb5kdc[31189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706731, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 18:32:11 dist-test-slave-qcpv krb5kdc[31189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706731, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:32:11.539191 31832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:11.527147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44568 (local address 127.28.207.126:34381)
0929 18:32:11.527651 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:11.527659 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:11.527736 (+ 77us) server_negotiation.cc:407] Connection header received
0929 18:32:11.527978 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:11.527982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:11.528034 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:11.528143 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:11.529099 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:11.529720 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:11.530807 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:11.531134 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:11.531260 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:11.534785 (+ 3525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:11.534790 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:11.534792 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:11.534835 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:11.536696 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:11.537365 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:11.537368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:11.537370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:11.537468 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:11.537967 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:11.537970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:11.537971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:11.538256 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:11.538402 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:11.538796 (+ 394us) server_negotiation.cc:299] Negotiation successful
0929 18:32:11.539021 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":243,"threads_started":1}
I20260929 18:32:11.543147 31234 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44568:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:32:11.543804 31234 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:32:11.550031 31511 tablet_service.cc:1467] Processing CreateTablet for tablet 92aaae8aa439412a8a8e23f96ba564c9 (DEFAULT_TABLE table=test-table [id=1bfbb4ab552a4d30a9e1aebb8bb739f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:11.550081 31650 tablet_service.cc:1467] Processing CreateTablet for tablet 92aaae8aa439412a8a8e23f96ba564c9 (DEFAULT_TABLE table=test-table [id=1bfbb4ab552a4d30a9e1aebb8bb739f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:11.550055 31371 tablet_service.cc:1467] Processing CreateTablet for tablet 92aaae8aa439412a8a8e23f96ba564c9 (DEFAULT_TABLE table=test-table [id=1bfbb4ab552a4d30a9e1aebb8bb739f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:11.550379 31650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92aaae8aa439412a8a8e23f96ba564c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:11.550379 31511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92aaae8aa439412a8a8e23f96ba564c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:11.550455 31371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92aaae8aa439412a8a8e23f96ba564c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:11.553164 31833 tablet_bootstrap.cc:492] T 92aaae8aa439412a8a8e23f96ba564c9 P 50a58713495847f580c2a4e37fc4fd2b: Bootstrap starting.
I20260929 18:32:11.553471 31834 tablet_bootstrap.cc:492] T 92aaae8aa439412a8a8e23f96ba564c9 P e2b2ec819b274e2a94fdeca8a004a587: Bootstrap starting.
I20260929 18:32:11.553546 31835 tablet_bootstrap.cc:492] T 92aaae8aa439412a8a8e23f96ba564c9 P 1b4afce4018d4a93b66adbfd4b1f5e49: Bootstrap starting.
I20260929 18:32:11.554600 31835 tablet_bootstrap.cc:654] T 92aaae8aa439412a8a8e23f96ba564c9 P 1b4afce4018d4a93b66adbfd4b1f5e49: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:11.555267 31834 tablet_bootstrap.cc:654] T 92aaae8aa439412a8a8e23f96ba564c9 P e2b2ec819b274e2a94fdeca8a004a587: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:11.555948 31833 tablet_bootstrap.cc:654] T 92aaae8aa439412a8a8e23f96ba564c9 P 50a58713495847f580c2a4e37fc4fd2b: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:11.556053 31835 tablet_bootstrap.cc:492] T 92aaae8aa439412a8a8e23f96ba564c9 P 1b4afce4018d4a93b66adbfd4b1f5e49: No bootstrap required, opened a new log
I20260929 18:32:11.556152 31835 ts_tablet_manager.cc:1397] T 92aaae8aa439412a8a8e23f96ba564c9 P 1b4afce4018d4a93b66adbfd4b1f5e49: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260929 18:32:11.556649 31835 raft_consensus.cc:348] T 92aaae8aa439412a8a8e23f96ba564c9 P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } }
I20260929 18:32:11.556806 31835 raft_consensus.cc:374] T 92aaae8aa439412a8a8e23f96ba564c9 P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:11.556850 31835 raft_consensus.cc:729] T 92aaae8aa439412a8a8e23f96ba564c9 P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b4afce4018d4a93b66adbfd4b1f5e49, State: Initialized, Role: FOLLOWER
I20260929 18:32:11.557027 31835 consensus_queue.cc:260] T 92aaae8aa439412a8a8e23f96ba564c9 P 1b4afce4018d4a93b66adbfd4b1f5e49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } }
I20260929 18:32:11.557344 31834 tablet_bootstrap.cc:492] T 92aaae8aa439412a8a8e23f96ba564c9 P e2b2ec819b274e2a94fdeca8a004a587: No bootstrap required, opened a new log
I20260929 18:32:11.557446 31834 ts_tablet_manager.cc:1397] T 92aaae8aa439412a8a8e23f96ba564c9 P e2b2ec819b274e2a94fdeca8a004a587: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 18:32:11.557456 31835 ts_tablet_manager.cc:1428] T 92aaae8aa439412a8a8e23f96ba564c9 P 1b4afce4018d4a93b66adbfd4b1f5e49: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260929 18:32:11.557520 31833 tablet_bootstrap.cc:492] T 92aaae8aa439412a8a8e23f96ba564c9 P 50a58713495847f580c2a4e37fc4fd2b: No bootstrap required, opened a new log
I20260929 18:32:11.557614 31833 ts_tablet_manager.cc:1397] T 92aaae8aa439412a8a8e23f96ba564c9 P 50a58713495847f580c2a4e37fc4fd2b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260929 18:32:11.557943 31834 raft_consensus.cc:348] T 92aaae8aa439412a8a8e23f96ba564c9 P e2b2ec819b274e2a94fdeca8a004a587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } }
I20260929 18:32:11.558070 31834 raft_consensus.cc:374] T 92aaae8aa439412a8a8e23f96ba564c9 P e2b2ec819b274e2a94fdeca8a004a587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:11.558111 31834 raft_consensus.cc:729] T 92aaae8aa439412a8a8e23f96ba564c9 P e2b2ec819b274e2a94fdeca8a004a587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2b2ec819b274e2a94fdeca8a004a587, State: Initialized, Role: FOLLOWER
I20260929 18:32:11.558162 31833 raft_consensus.cc:348] T 92aaae8aa439412a8a8e23f96ba564c9 P 50a58713495847f580c2a4e37fc4fd2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } }
I20260929 18:32:11.558295 31833 raft_consensus.cc:374] T 92aaae8aa439412a8a8e23f96ba564c9 P 50a58713495847f580c2a4e37fc4fd2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:11.558341 31833 raft_consensus.cc:729] T 92aaae8aa439412a8a8e23f96ba564c9 P 50a58713495847f580c2a4e37fc4fd2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50a58713495847f580c2a4e37fc4fd2b, State: Initialized, Role: FOLLOWER
I20260929 18:32:11.558255 31834 consensus_queue.cc:260] T 92aaae8aa439412a8a8e23f96ba564c9 P e2b2ec819b274e2a94fdeca8a004a587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } }
I20260929 18:32:11.558476 31833 consensus_queue.cc:260] T 92aaae8aa439412a8a8e23f96ba564c9 P 50a58713495847f580c2a4e37fc4fd2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } }
I20260929 18:32:11.558737 31834 ts_tablet_manager.cc:1428] T 92aaae8aa439412a8a8e23f96ba564c9 P e2b2ec819b274e2a94fdeca8a004a587: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 18:32:11.558782 31833 ts_tablet_manager.cc:1428] T 92aaae8aa439412a8a8e23f96ba564c9 P 50a58713495847f580c2a4e37fc4fd2b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 18:32:11.598829 31817 raft_consensus.cc:482] T 92aaae8aa439412a8a8e23f96ba564c9 P 50a58713495847f580c2a4e37fc4fd2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:32:11.599148 31817 raft_consensus.cc:504] T 92aaae8aa439412a8a8e23f96ba564c9 P 50a58713495847f580c2a4e37fc4fd2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } }
I20260929 18:32:11.599704 31817 leader_election.cc:288] T 92aaae8aa439412a8a8e23f96ba564c9 P 50a58713495847f580c2a4e37fc4fd2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b4afce4018d4a93b66adbfd4b1f5e49 (127.28.207.65:46387), e2b2ec819b274e2a94fdeca8a004a587 (127.28.207.66:41597)
I20260929 18:32:11.600260 31391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92aaae8aa439412a8a8e23f96ba564c9" candidate_uuid: "50a58713495847f580c2a4e37fc4fd2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" is_pre_election: true
I20260929 18:32:11.600479 31391 raft_consensus.cc:2454] T 92aaae8aa439412a8a8e23f96ba564c9 P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50a58713495847f580c2a4e37fc4fd2b in term 0.
I20260929 18:32:11.600400 31524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92aaae8aa439412a8a8e23f96ba564c9" candidate_uuid: "50a58713495847f580c2a4e37fc4fd2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2b2ec819b274e2a94fdeca8a004a587" is_pre_election: true
I20260929 18:32:11.600608 31524 raft_consensus.cc:2454] T 92aaae8aa439412a8a8e23f96ba564c9 P e2b2ec819b274e2a94fdeca8a004a587 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50a58713495847f580c2a4e37fc4fd2b in term 0.
I20260929 18:32:11.600884 31604 leader_election.cc:302] T 92aaae8aa439412a8a8e23f96ba564c9 P 50a58713495847f580c2a4e37fc4fd2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b4afce4018d4a93b66adbfd4b1f5e49, 50a58713495847f580c2a4e37fc4fd2b; no voters:
I20260929 18:32:11.601161 31817 raft_consensus.cc:2781] T 92aaae8aa439412a8a8e23f96ba564c9 P 50a58713495847f580c2a4e37fc4fd2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:32:11.601244 31817 raft_consensus.cc:482] T 92aaae8aa439412a8a8e23f96ba564c9 P 50a58713495847f580c2a4e37fc4fd2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:32:11.601276 31817 raft_consensus.cc:3037] T 92aaae8aa439412a8a8e23f96ba564c9 P 50a58713495847f580c2a4e37fc4fd2b [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:11.602377 31817 raft_consensus.cc:504] T 92aaae8aa439412a8a8e23f96ba564c9 P 50a58713495847f580c2a4e37fc4fd2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } }
I20260929 18:32:11.602780 31817 leader_election.cc:288] T 92aaae8aa439412a8a8e23f96ba564c9 P 50a58713495847f580c2a4e37fc4fd2b [CANDIDATE]: Term 1 election: Requested vote from peers 1b4afce4018d4a93b66adbfd4b1f5e49 (127.28.207.65:46387), e2b2ec819b274e2a94fdeca8a004a587 (127.28.207.66:41597)
I20260929 18:32:11.603330 31391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92aaae8aa439412a8a8e23f96ba564c9" candidate_uuid: "50a58713495847f580c2a4e37fc4fd2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49"
I20260929 18:32:11.603484 31391 raft_consensus.cc:3037] T 92aaae8aa439412a8a8e23f96ba564c9 P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:11.604159 31524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92aaae8aa439412a8a8e23f96ba564c9" candidate_uuid: "50a58713495847f580c2a4e37fc4fd2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2b2ec819b274e2a94fdeca8a004a587"
I20260929 18:32:11.604310 31524 raft_consensus.cc:3037] T 92aaae8aa439412a8a8e23f96ba564c9 P e2b2ec819b274e2a94fdeca8a004a587 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:11.605059 31391 raft_consensus.cc:2454] T 92aaae8aa439412a8a8e23f96ba564c9 P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50a58713495847f580c2a4e37fc4fd2b in term 1.
I20260929 18:32:11.605464 31604 leader_election.cc:302] T 92aaae8aa439412a8a8e23f96ba564c9 P 50a58713495847f580c2a4e37fc4fd2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b4afce4018d4a93b66adbfd4b1f5e49, 50a58713495847f580c2a4e37fc4fd2b; no voters:
I20260929 18:32:11.605695 31817 raft_consensus.cc:2781] T 92aaae8aa439412a8a8e23f96ba564c9 P 50a58713495847f580c2a4e37fc4fd2b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:32:11.605815 31817 raft_consensus.cc:686] T 92aaae8aa439412a8a8e23f96ba564c9 P 50a58713495847f580c2a4e37fc4fd2b [term 1 LEADER]: Becoming Leader. State: Replica: 50a58713495847f580c2a4e37fc4fd2b, State: Running, Role: LEADER
I20260929 18:32:11.605808 31524 raft_consensus.cc:2454] T 92aaae8aa439412a8a8e23f96ba564c9 P e2b2ec819b274e2a94fdeca8a004a587 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50a58713495847f580c2a4e37fc4fd2b in term 1.
I20260929 18:32:11.606001 31817 consensus_queue.cc:237] T 92aaae8aa439412a8a8e23f96ba564c9 P 50a58713495847f580c2a4e37fc4fd2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } }
I20260929 18:32:11.608285 31229 catalog_manager.cc:5495] T 92aaae8aa439412a8a8e23f96ba564c9 P 50a58713495847f580c2a4e37fc4fd2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 50a58713495847f580c2a4e37fc4fd2b (127.28.207.67). New cstate: current_term: 1 leader_uuid: "50a58713495847f580c2a4e37fc4fd2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } health_report { overall_health: HEALTHY } } }
I20260929 18:32:11.637756 31842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:11.633556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52958 (local address 127.28.207.67:34053)
0929 18:32:11.633931 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:11.633940 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:11.633955 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:32:11.634004 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:11.634006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:11.634058 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:11.634156 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:32:11.634501 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:11.635365 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:11.636269 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:11.636533 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:11.636650 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:11.637045 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:32:11.637138 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:32:11.637212 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:11.637466 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 18:32:11.637598 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":141,"threads_started":1}
I20260929 18:32:11.645859 31524 raft_consensus.cc:1261] T 92aaae8aa439412a8a8e23f96ba564c9 P e2b2ec819b274e2a94fdeca8a004a587 [term 1 FOLLOWER]: Refusing update from remote peer 50a58713495847f580c2a4e37fc4fd2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:32:11.645993 31391 raft_consensus.cc:1261] T 92aaae8aa439412a8a8e23f96ba564c9 P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 1 FOLLOWER]: Refusing update from remote peer 50a58713495847f580c2a4e37fc4fd2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:32:11.646504 31803 consensus_queue.cc:1035] T 92aaae8aa439412a8a8e23f96ba564c9 P 50a58713495847f580c2a4e37fc4fd2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:32:11.646768 31778 consensus_queue.cc:1035] T 92aaae8aa439412a8a8e23f96ba564c9 P 50a58713495847f580c2a4e37fc4fd2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:32:11.650596 31805 mvcc.cc:205] Tried to move back new op lower bound from 7334734772818034688 to 7334734772660101120. Current Snapshot: MvccSnapshot[applied={T|T < 7334734772818034688}]
I20260929 18:32:11.685542 31231 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:32:11.685719 31231 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:32:11.688256 31231 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d27777167194a44a0b5510f5b8d6c99: Sending DeleteTablet for 3 replicas of tablet 92aaae8aa439412a8a8e23f96ba564c9
I20260929 18:32:11.689078 31511 tablet_service.cc:1514] Processing DeleteTablet for tablet 92aaae8aa439412a8a8e23f96ba564c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:11 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:41982
I20260929 18:32:11.689193 31371 tablet_service.cc:1514] Processing DeleteTablet for tablet 92aaae8aa439412a8a8e23f96ba564c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:11 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:43920
I20260929 18:32:11.689234 31650 tablet_service.cc:1514] Processing DeleteTablet for tablet 92aaae8aa439412a8a8e23f96ba564c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:11 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:52934
I20260929 18:32:11.689337 31822 tablet_replica.cc:330] T 92aaae8aa439412a8a8e23f96ba564c9 P e2b2ec819b274e2a94fdeca8a004a587: stopping tablet replica
I20260929 18:32:11.689483 31822 raft_consensus.cc:2229] T 92aaae8aa439412a8a8e23f96ba564c9 P e2b2ec819b274e2a94fdeca8a004a587 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:32:11.689567 31823 tablet_replica.cc:330] T 92aaae8aa439412a8a8e23f96ba564c9 P 1b4afce4018d4a93b66adbfd4b1f5e49: stopping tablet replica
I20260929 18:32:11.689587 31822 raft_consensus.cc:2258] T 92aaae8aa439412a8a8e23f96ba564c9 P e2b2ec819b274e2a94fdeca8a004a587 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:11.689726 31823 raft_consensus.cc:2229] T 92aaae8aa439412a8a8e23f96ba564c9 P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:32:11.689888 31823 raft_consensus.cc:2258] T 92aaae8aa439412a8a8e23f96ba564c9 P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:11.690124 31824 tablet_replica.cc:330] T 92aaae8aa439412a8a8e23f96ba564c9 P 50a58713495847f580c2a4e37fc4fd2b: stopping tablet replica
I20260929 18:32:11.690289 31824 raft_consensus.cc:2229] T 92aaae8aa439412a8a8e23f96ba564c9 P 50a58713495847f580c2a4e37fc4fd2b [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:32:11.690296 31822 ts_tablet_manager.cc:1905] T 92aaae8aa439412a8a8e23f96ba564c9 P e2b2ec819b274e2a94fdeca8a004a587: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:32:11.690296 31231 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44568:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 18:32:11.690537 31824 raft_consensus.cc:2258] T 92aaae8aa439412a8a8e23f96ba564c9 P 50a58713495847f580c2a4e37fc4fd2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:11.690778 31823 ts_tablet_manager.cc:1905] T 92aaae8aa439412a8a8e23f96ba564c9 P 1b4afce4018d4a93b66adbfd4b1f5e49: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 18:32:11.690785 31231 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:32:11.691818 31824 ts_tablet_manager.cc:1905] T 92aaae8aa439412a8a8e23f96ba564c9 P 50a58713495847f580c2a4e37fc4fd2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:32:11.692813 31822 ts_tablet_manager.cc:1918] T 92aaae8aa439412a8a8e23f96ba564c9 P e2b2ec819b274e2a94fdeca8a004a587: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:32:11.692914 31822 log.cc:1196] T 92aaae8aa439412a8a8e23f96ba564c9 P e2b2ec819b274e2a94fdeca8a004a587: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-1/wal/wals/92aaae8aa439412a8a8e23f96ba564c9
I20260929 18:32:11.693305 31822 ts_tablet_manager.cc:1939] T 92aaae8aa439412a8a8e23f96ba564c9 P e2b2ec819b274e2a94fdeca8a004a587: Deleting consensus metadata
I20260929 18:32:11.694298 31221 catalog_manager.cc:4841] TS e2b2ec819b274e2a94fdeca8a004a587 (127.28.207.66:41597): tablet 92aaae8aa439412a8a8e23f96ba564c9 (table test-table [id=1bfbb4ab552a4d30a9e1aebb8bb739f6]) successfully deleted
I20260929 18:32:11.694567 31823 ts_tablet_manager.cc:1918] T 92aaae8aa439412a8a8e23f96ba564c9 P 1b4afce4018d4a93b66adbfd4b1f5e49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:32:11.694667 31823 log.cc:1196] T 92aaae8aa439412a8a8e23f96ba564c9 P 1b4afce4018d4a93b66adbfd4b1f5e49: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-0/wal/wals/92aaae8aa439412a8a8e23f96ba564c9
I20260929 18:32:11.694983 31823 ts_tablet_manager.cc:1939] T 92aaae8aa439412a8a8e23f96ba564c9 P 1b4afce4018d4a93b66adbfd4b1f5e49: Deleting consensus metadata
I20260929 18:32:11.695886 31220 catalog_manager.cc:4841] TS 1b4afce4018d4a93b66adbfd4b1f5e49 (127.28.207.65:46387): tablet 92aaae8aa439412a8a8e23f96ba564c9 (table test-table [id=1bfbb4ab552a4d30a9e1aebb8bb739f6]) successfully deleted
I20260929 18:32:11.696115 31824 ts_tablet_manager.cc:1918] T 92aaae8aa439412a8a8e23f96ba564c9 P 50a58713495847f580c2a4e37fc4fd2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:32:11.696236 31824 log.cc:1196] T 92aaae8aa439412a8a8e23f96ba564c9 P 50a58713495847f580c2a4e37fc4fd2b: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-2/wal/wals/92aaae8aa439412a8a8e23f96ba564c9
I20260929 18:32:11.696551 31824 ts_tablet_manager.cc:1939] T 92aaae8aa439412a8a8e23f96ba564c9 P 50a58713495847f580c2a4e37fc4fd2b: Deleting consensus metadata
I20260929 18:32:11.697485 31220 catalog_manager.cc:4841] TS 50a58713495847f580c2a4e37fc4fd2b (127.28.207.67:34053): tablet 92aaae8aa439412a8a8e23f96ba564c9 (table test-table [id=1bfbb4ab552a4d30a9e1aebb8bb739f6]) successfully deleted
I20260929 18:32:11.699266 31371 tablet_service.cc:1467] Processing CreateTablet for tablet 3174af4edd534194aa7218c8d0784b22 (DEFAULT_TABLE table=test-table [id=d79015b80a0e4839ad0237b5990bea2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:11.699335 31650 tablet_service.cc:1467] Processing CreateTablet for tablet 3174af4edd534194aa7218c8d0784b22 (DEFAULT_TABLE table=test-table [id=d79015b80a0e4839ad0237b5990bea2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:11.699347 31511 tablet_service.cc:1467] Processing CreateTablet for tablet 3174af4edd534194aa7218c8d0784b22 (DEFAULT_TABLE table=test-table [id=d79015b80a0e4839ad0237b5990bea2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:11.699631 31650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3174af4edd534194aa7218c8d0784b22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:11.699631 31371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3174af4edd534194aa7218c8d0784b22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:11.699710 31511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3174af4edd534194aa7218c8d0784b22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:11.701822 31834 tablet_bootstrap.cc:492] T 3174af4edd534194aa7218c8d0784b22 P e2b2ec819b274e2a94fdeca8a004a587: Bootstrap starting.
I20260929 18:32:11.702577 31833 tablet_bootstrap.cc:492] T 3174af4edd534194aa7218c8d0784b22 P 50a58713495847f580c2a4e37fc4fd2b: Bootstrap starting.
I20260929 18:32:11.702759 31835 tablet_bootstrap.cc:492] T 3174af4edd534194aa7218c8d0784b22 P 1b4afce4018d4a93b66adbfd4b1f5e49: Bootstrap starting.
I20260929 18:32:11.702991 31834 tablet_bootstrap.cc:654] T 3174af4edd534194aa7218c8d0784b22 P e2b2ec819b274e2a94fdeca8a004a587: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:11.703657 31835 tablet_bootstrap.cc:654] T 3174af4edd534194aa7218c8d0784b22 P 1b4afce4018d4a93b66adbfd4b1f5e49: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:11.703657 31833 tablet_bootstrap.cc:654] T 3174af4edd534194aa7218c8d0784b22 P 50a58713495847f580c2a4e37fc4fd2b: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:11.704207 31834 tablet_bootstrap.cc:492] T 3174af4edd534194aa7218c8d0784b22 P e2b2ec819b274e2a94fdeca8a004a587: No bootstrap required, opened a new log
I20260929 18:32:11.704298 31834 ts_tablet_manager.cc:1397] T 3174af4edd534194aa7218c8d0784b22 P e2b2ec819b274e2a94fdeca8a004a587: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260929 18:32:11.704798 31834 raft_consensus.cc:348] T 3174af4edd534194aa7218c8d0784b22 P e2b2ec819b274e2a94fdeca8a004a587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } }
I20260929 18:32:11.704946 31834 raft_consensus.cc:374] T 3174af4edd534194aa7218c8d0784b22 P e2b2ec819b274e2a94fdeca8a004a587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:11.704984 31834 raft_consensus.cc:729] T 3174af4edd534194aa7218c8d0784b22 P e2b2ec819b274e2a94fdeca8a004a587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2b2ec819b274e2a94fdeca8a004a587, State: Initialized, Role: FOLLOWER
I20260929 18:32:11.705315 31834 consensus_queue.cc:260] T 3174af4edd534194aa7218c8d0784b22 P e2b2ec819b274e2a94fdeca8a004a587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } }
I20260929 18:32:11.705430 31835 tablet_bootstrap.cc:492] T 3174af4edd534194aa7218c8d0784b22 P 1b4afce4018d4a93b66adbfd4b1f5e49: No bootstrap required, opened a new log
I20260929 18:32:11.705535 31835 ts_tablet_manager.cc:1397] T 3174af4edd534194aa7218c8d0784b22 P 1b4afce4018d4a93b66adbfd4b1f5e49: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 18:32:11.705596 31833 tablet_bootstrap.cc:492] T 3174af4edd534194aa7218c8d0784b22 P 50a58713495847f580c2a4e37fc4fd2b: No bootstrap required, opened a new log
I20260929 18:32:11.705706 31833 ts_tablet_manager.cc:1397] T 3174af4edd534194aa7218c8d0784b22 P 50a58713495847f580c2a4e37fc4fd2b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:32:11.706036 31835 raft_consensus.cc:348] T 3174af4edd534194aa7218c8d0784b22 P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } }
I20260929 18:32:11.706143 31834 ts_tablet_manager.cc:1428] T 3174af4edd534194aa7218c8d0784b22 P e2b2ec819b274e2a94fdeca8a004a587: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260929 18:32:11.706177 31835 raft_consensus.cc:374] T 3174af4edd534194aa7218c8d0784b22 P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:11.706146 31833 raft_consensus.cc:348] T 3174af4edd534194aa7218c8d0784b22 P 50a58713495847f580c2a4e37fc4fd2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } }
I20260929 18:32:11.706219 31835 raft_consensus.cc:729] T 3174af4edd534194aa7218c8d0784b22 P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b4afce4018d4a93b66adbfd4b1f5e49, State: Initialized, Role: FOLLOWER
I20260929 18:32:11.706259 31833 raft_consensus.cc:374] T 3174af4edd534194aa7218c8d0784b22 P 50a58713495847f580c2a4e37fc4fd2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:11.706293 31833 raft_consensus.cc:729] T 3174af4edd534194aa7218c8d0784b22 P 50a58713495847f580c2a4e37fc4fd2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50a58713495847f580c2a4e37fc4fd2b, State: Initialized, Role: FOLLOWER
I20260929 18:32:11.706364 31835 consensus_queue.cc:260] T 3174af4edd534194aa7218c8d0784b22 P 1b4afce4018d4a93b66adbfd4b1f5e49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } }
I20260929 18:32:11.706425 31833 consensus_queue.cc:260] T 3174af4edd534194aa7218c8d0784b22 P 50a58713495847f580c2a4e37fc4fd2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } }
I20260929 18:32:11.706713 31833 ts_tablet_manager.cc:1428] T 3174af4edd534194aa7218c8d0784b22 P 50a58713495847f580c2a4e37fc4fd2b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 18:32:11.707145 31835 ts_tablet_manager.cc:1428] T 3174af4edd534194aa7218c8d0784b22 P 1b4afce4018d4a93b66adbfd4b1f5e49: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:32:11.720515 31839 raft_consensus.cc:482] T 3174af4edd534194aa7218c8d0784b22 P e2b2ec819b274e2a94fdeca8a004a587 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:32:11.720645 31839 raft_consensus.cc:504] T 3174af4edd534194aa7218c8d0784b22 P e2b2ec819b274e2a94fdeca8a004a587 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } }
I20260929 18:32:11.721973 31839 leader_election.cc:288] T 3174af4edd534194aa7218c8d0784b22 P e2b2ec819b274e2a94fdeca8a004a587 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b4afce4018d4a93b66adbfd4b1f5e49 (127.28.207.65:46387), 50a58713495847f580c2a4e37fc4fd2b (127.28.207.67:34053)
I20260929 18:32:11.728236 31850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:11.722559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:34053 (local address 127.28.207.66:56919)
0929 18:32:11.722913 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:11.722929 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:11.723032 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:11.723430 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:11.723432 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:11.723447 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:11.723603 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:11.723610 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:11.724863 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:11.724865 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:11.726508 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:11.726518 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:11.727749 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:11.727753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:11.727851 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:11.727879 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 18:32:11.727961 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 18:32:11.728040 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":117,"thread_start_us":116,"threads_started":1}
I20260929 18:32:11.728411 31842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:11.722962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:56919 (local address 127.28.207.67:34053)
0929 18:32:11.723153 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:11.723158 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:11.723176 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:32:11.723245 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:11.723248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:11.723307 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:11.723407 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:11.723813 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:11.724722 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:11.726675 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:11.727393 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:11.727563 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:11.727631 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:11.728128 (+ 497us) server_negotiation.cc:299] Negotiation successful
0929 18:32:11.728272 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 18:32:11.728768 31671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3174af4edd534194aa7218c8d0784b22" candidate_uuid: "e2b2ec819b274e2a94fdeca8a004a587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50a58713495847f580c2a4e37fc4fd2b" is_pre_election: true
I20260929 18:32:11.728845 31849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:11.722177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:46387 (local address 127.28.207.66:46545)
0929 18:32:11.722504 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:11.722547 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:11.722738 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:11.723561 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:11.723566 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:11.723582 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:11.723725 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:11.723733 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:11.724860 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:11.724863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:11.726508 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:11.726518 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:11.728430 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:11.728434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:11.728522 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:11.728549 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 18:32:11.728634 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 18:32:11.728692 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":207,"thread_start_us":97,"threads_started":1}
I20260929 18:32:11.728935 31671 raft_consensus.cc:2454] T 3174af4edd534194aa7218c8d0784b22 P 50a58713495847f580c2a4e37fc4fd2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2b2ec819b274e2a94fdeca8a004a587 in term 0.
I20260929 18:32:11.729642 31464 leader_election.cc:302] T 3174af4edd534194aa7218c8d0784b22 P e2b2ec819b274e2a94fdeca8a004a587 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50a58713495847f580c2a4e37fc4fd2b, e2b2ec819b274e2a94fdeca8a004a587; no voters:
I20260929 18:32:11.729812 31851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:11.722980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:46545 (local address 127.28.207.65:46387)
0929 18:32:11.723287 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:11.723303 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:11.723319 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:32:11.723378 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:11.723383 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:11.723437 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:11.723535 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:11.723860 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:11.724699 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:11.726675 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:11.727413 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:11.727563 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:11.727631 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:11.729407 (+ 1776us) server_negotiation.cc:299] Negotiation successful
0929 18:32:11.729626 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":102,"threads_started":1}
I20260929 18:32:11.729975 31839 raft_consensus.cc:2781] T 3174af4edd534194aa7218c8d0784b22 P e2b2ec819b274e2a94fdeca8a004a587 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:32:11.730046 31839 raft_consensus.cc:482] T 3174af4edd534194aa7218c8d0784b22 P e2b2ec819b274e2a94fdeca8a004a587 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:32:11.730085 31839 raft_consensus.cc:3037] T 3174af4edd534194aa7218c8d0784b22 P e2b2ec819b274e2a94fdeca8a004a587 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:11.730370 31391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3174af4edd534194aa7218c8d0784b22" candidate_uuid: "e2b2ec819b274e2a94fdeca8a004a587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" is_pre_election: true
I20260929 18:32:11.730521 31391 raft_consensus.cc:2454] T 3174af4edd534194aa7218c8d0784b22 P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2b2ec819b274e2a94fdeca8a004a587 in term 0.
I20260929 18:32:11.731832 31839 raft_consensus.cc:504] T 3174af4edd534194aa7218c8d0784b22 P e2b2ec819b274e2a94fdeca8a004a587 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } }
I20260929 18:32:11.732223 31839 leader_election.cc:288] T 3174af4edd534194aa7218c8d0784b22 P e2b2ec819b274e2a94fdeca8a004a587 [CANDIDATE]: Term 1 election: Requested vote from peers 1b4afce4018d4a93b66adbfd4b1f5e49 (127.28.207.65:46387), 50a58713495847f580c2a4e37fc4fd2b (127.28.207.67:34053)
I20260929 18:32:11.732901 31391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3174af4edd534194aa7218c8d0784b22" candidate_uuid: "e2b2ec819b274e2a94fdeca8a004a587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49"
I20260929 18:32:11.733071 31391 raft_consensus.cc:3037] T 3174af4edd534194aa7218c8d0784b22 P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:11.733206 31671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3174af4edd534194aa7218c8d0784b22" candidate_uuid: "e2b2ec819b274e2a94fdeca8a004a587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50a58713495847f580c2a4e37fc4fd2b"
I20260929 18:32:11.733381 31671 raft_consensus.cc:3037] T 3174af4edd534194aa7218c8d0784b22 P 50a58713495847f580c2a4e37fc4fd2b [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:11.734258 31391 raft_consensus.cc:2454] T 3174af4edd534194aa7218c8d0784b22 P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2b2ec819b274e2a94fdeca8a004a587 in term 1.
I20260929 18:32:11.734586 31464 leader_election.cc:302] T 3174af4edd534194aa7218c8d0784b22 P e2b2ec819b274e2a94fdeca8a004a587 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b4afce4018d4a93b66adbfd4b1f5e49, e2b2ec819b274e2a94fdeca8a004a587; no voters:
I20260929 18:32:11.734793 31839 raft_consensus.cc:2781] T 3174af4edd534194aa7218c8d0784b22 P e2b2ec819b274e2a94fdeca8a004a587 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:32:11.734925 31671 raft_consensus.cc:2454] T 3174af4edd534194aa7218c8d0784b22 P 50a58713495847f580c2a4e37fc4fd2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2b2ec819b274e2a94fdeca8a004a587 in term 1.
I20260929 18:32:11.735105 31839 raft_consensus.cc:686] T 3174af4edd534194aa7218c8d0784b22 P e2b2ec819b274e2a94fdeca8a004a587 [term 1 LEADER]: Becoming Leader. State: Replica: e2b2ec819b274e2a94fdeca8a004a587, State: Running, Role: LEADER
I20260929 18:32:11.735419 31839 consensus_queue.cc:237] T 3174af4edd534194aa7218c8d0784b22 P e2b2ec819b274e2a94fdeca8a004a587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } }
I20260929 18:32:11.738154 31234 catalog_manager.cc:5495] T 3174af4edd534194aa7218c8d0784b22 P e2b2ec819b274e2a94fdeca8a004a587 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2b2ec819b274e2a94fdeca8a004a587 (127.28.207.66). New cstate: current_term: 1 leader_uuid: "e2b2ec819b274e2a94fdeca8a004a587" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2b2ec819b274e2a94fdeca8a004a587" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 41597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 } health_report { overall_health: UNKNOWN } } }
I20260929 18:32:11.759690 31854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:11.755741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41990 (local address 127.28.207.66:41597)
0929 18:32:11.756104 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:11.756111 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:11.756124 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:32:11.756198 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:11.756201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:11.756253 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:11.756390 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:32:11.756665 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:11.757213 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:11.758048 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:11.758288 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:11.758398 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:11.758670 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:32:11.759225 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:32:11.759360 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:11.759425 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 18:32:11.759484 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":161,"threads_started":1}
I20260929 18:32:11.773075 31842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:11.768916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:40019 (local address 127.28.207.67:34053)
0929 18:32:11.769222 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:11.769234 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:11.769353 (+ 119us) server_negotiation.cc:407] Connection header received
0929 18:32:11.769537 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:11.769566 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:11.769622 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:11.769708 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:32:11.770128 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:11.770732 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:11.771794 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:11.772002 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:11.772092 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:11.772446 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:32:11.772521 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:32:11.772577 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:11.772914 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 18:32:11.772964 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260929 18:32:11.773119 31857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:11.768696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:34053 (local address 127.28.207.66:40019)
0929 18:32:11.769220 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:11.769241 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:11.769415 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:11.769790 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:11.769792 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:11.769801 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:32:11.769954 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:11.769960 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:11.770923 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:11.770926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:11.771574 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:11.771583 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:11.772184 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:11.772187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:11.772261 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:11.772311 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:32:11.772682 (+ 371us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:32:11.772690 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 18:32:11.772903 (+ 213us) client_negotiation.cc:240] Negotiation successful
0929 18:32:11.772981 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":134,"threads_started":1}
I20260929 18:32:11.780886 31391 raft_consensus.cc:1261] T 3174af4edd534194aa7218c8d0784b22 P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 1 FOLLOWER]: Refusing update from remote peer e2b2ec819b274e2a94fdeca8a004a587: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:32:11.780896 31671 raft_consensus.cc:1261] T 3174af4edd534194aa7218c8d0784b22 P 50a58713495847f580c2a4e37fc4fd2b [term 1 FOLLOWER]: Refusing update from remote peer e2b2ec819b274e2a94fdeca8a004a587: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:32:11.781514 31852 consensus_queue.cc:1035] T 3174af4edd534194aa7218c8d0784b22 P e2b2ec819b274e2a94fdeca8a004a587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50a58713495847f580c2a4e37fc4fd2b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:32:11.781924 31839 consensus_queue.cc:1035] T 3174af4edd534194aa7218c8d0784b22 P e2b2ec819b274e2a94fdeca8a004a587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b4afce4018d4a93b66adbfd4b1f5e49" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:32:11.804354 31854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:11.799760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:39773 (local address 127.28.207.66:41597)
0929 18:32:11.800008 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:11.800013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:11.800147 (+ 134us) server_negotiation.cc:407] Connection header received
0929 18:32:11.800378 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:11.800381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:11.800445 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:11.800493 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:32:11.801051 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:11.801634 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:11.803009 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:11.803349 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:11.803426 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:11.803843 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:32:11.803940 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:32:11.804027 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:11.804135 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 18:32:11.804202 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260929 18:32:11.804404 31861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:11.799695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:41597 (local address 127.28.207.67:39773)
0929 18:32:11.800035 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:11.800057 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:11.800234 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:11.800735 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:11.800738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:11.800748 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:32:11.800891 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:11.800898 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:11.801770 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:11.801774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:11.802815 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:11.802825 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:11.803548 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:11.803552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:11.803639 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:11.803674 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:32:11.804048 (+ 374us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:32:11.804054 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:32:11.804119 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 18:32:11.804218 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":117,"threads_started":1}
W20260929 18:32:11.805236 31805 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 18:32:13.380757 31229 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:32:13.380931 31229 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:32:13.384349 31229 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d27777167194a44a0b5510f5b8d6c99: Sending DeleteTablet for 3 replicas of tablet 3174af4edd534194aa7218c8d0784b22
I20260929 18:32:13.385125 31371 tablet_service.cc:1514] Processing DeleteTablet for tablet 3174af4edd534194aa7218c8d0784b22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:13 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:43920
I20260929 18:32:13.385381 31511 tablet_service.cc:1514] Processing DeleteTablet for tablet 3174af4edd534194aa7218c8d0784b22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:13 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:41982
I20260929 18:32:13.385717 31886 tablet_replica.cc:330] T 3174af4edd534194aa7218c8d0784b22 P e2b2ec819b274e2a94fdeca8a004a587: stopping tablet replica
I20260929 18:32:13.385725 31650 tablet_service.cc:1514] Processing DeleteTablet for tablet 3174af4edd534194aa7218c8d0784b22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:13 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:52934
I20260929 18:32:13.385731 31885 tablet_replica.cc:330] T 3174af4edd534194aa7218c8d0784b22 P 1b4afce4018d4a93b66adbfd4b1f5e49: stopping tablet replica
I20260929 18:32:13.385857 31886 raft_consensus.cc:2229] T 3174af4edd534194aa7218c8d0784b22 P e2b2ec819b274e2a94fdeca8a004a587 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:32:13.385939 31885 raft_consensus.cc:2229] T 3174af4edd534194aa7218c8d0784b22 P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:32:13.386123 31885 raft_consensus.cc:2258] T 3174af4edd534194aa7218c8d0784b22 P 1b4afce4018d4a93b66adbfd4b1f5e49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:13.386129 31887 tablet_replica.cc:330] T 3174af4edd534194aa7218c8d0784b22 P 50a58713495847f580c2a4e37fc4fd2b: stopping tablet replica
I20260929 18:32:13.386225 31886 raft_consensus.cc:2258] T 3174af4edd534194aa7218c8d0784b22 P e2b2ec819b274e2a94fdeca8a004a587 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:13.386318 31887 raft_consensus.cc:2229] T 3174af4edd534194aa7218c8d0784b22 P 50a58713495847f580c2a4e37fc4fd2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:32:13.386412 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 31306
I20260929 18:32:13.386462 31887 raft_consensus.cc:2258] T 3174af4edd534194aa7218c8d0784b22 P 50a58713495847f580c2a4e37fc4fd2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:13.388110 31886 ts_tablet_manager.cc:1905] T 3174af4edd534194aa7218c8d0784b22 P e2b2ec819b274e2a94fdeca8a004a587: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:32:13.388110 31887 ts_tablet_manager.cc:1905] T 3174af4edd534194aa7218c8d0784b22 P 50a58713495847f580c2a4e37fc4fd2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:32:13.391338 31886 ts_tablet_manager.cc:1918] T 3174af4edd534194aa7218c8d0784b22 P e2b2ec819b274e2a94fdeca8a004a587: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 18:32:13.391338 31887 ts_tablet_manager.cc:1918] T 3174af4edd534194aa7218c8d0784b22 P 50a58713495847f580c2a4e37fc4fd2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 18:32:13.391434 31887 log.cc:1196] T 3174af4edd534194aa7218c8d0784b22 P 50a58713495847f580c2a4e37fc4fd2b: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-2/wal/wals/3174af4edd534194aa7218c8d0784b22
I20260929 18:32:13.391434 31886 log.cc:1196] T 3174af4edd534194aa7218c8d0784b22 P e2b2ec819b274e2a94fdeca8a004a587: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790706718260899-29501-0/minicluster-data/ts-1/wal/wals/3174af4edd534194aa7218c8d0784b22
I20260929 18:32:13.391728 31887 ts_tablet_manager.cc:1939] T 3174af4edd534194aa7218c8d0784b22 P 50a58713495847f580c2a4e37fc4fd2b: Deleting consensus metadata
I20260929 18:32:13.391737 31886 ts_tablet_manager.cc:1939] T 3174af4edd534194aa7218c8d0784b22 P e2b2ec819b274e2a94fdeca8a004a587: Deleting consensus metadata
I20260929 18:32:13.392989 31221 catalog_manager.cc:4841] TS e2b2ec819b274e2a94fdeca8a004a587 (127.28.207.66:41597): tablet 3174af4edd534194aa7218c8d0784b22 (table test-table [id=d79015b80a0e4839ad0237b5990bea2e]) successfully deleted
I20260929 18:32:13.395035 31220 catalog_manager.cc:4841] TS 50a58713495847f580c2a4e37fc4fd2b (127.28.207.67:34053): tablet 3174af4edd534194aa7218c8d0784b22 (table test-table [id=d79015b80a0e4839ad0237b5990bea2e]) successfully deleted
W20260929 18:32:13.396319 31220 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.207.65:46387 (error 108)
I20260929 18:32:13.396744 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 31446
I20260929 18:32:13.398622 31889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:13.398161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:46387 (local address 127.0.0.1:43934)
0929 18:32:13.398444 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:13.398528 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.65:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":98,"threads_started":1}
W20260929 18:32:13.398952 31220 catalog_manager.cc:4584] TS 1b4afce4018d4a93b66adbfd4b1f5e49 (127.28.207.65:46387): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3174af4edd534194aa7218c8d0784b22: Network error: Client connection negotiation failed: client connection to 127.28.207.65:46387: connect: Connection refused (error 111)
I20260929 18:32:13.405102 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 31586
I20260929 18:32:13.413012 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 31202
2026-09-29T18:32:13Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6728 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:32:13 dist-test-slave-qcpv krb5kdc[31892](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:32:13 dist-test-slave-qcpv krb5kdc[31892](info): set up 2 sockets
Sep 29 18:32:13 dist-test-slave-qcpv krb5kdc[31892](info): commencing operation
krb5kdc: starting...
W20260929 18:32:14.547313 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:32:14 dist-test-slave-qcpv krb5kdc[31892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:32:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:32:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.207.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
I20260929 18:32:14.687686 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:43405
--webserver_interface=127.28.207.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:42205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:43405
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:14.826560 31905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:14.827088 31905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:14.827183 31905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:14.831768 31905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:32:14.831887 31905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:14.831925 31905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:14.831964 31905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:32:14.832022 31905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:32:14.838708 31905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:42205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:43405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:43405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:14.840387 31905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:14.841709 31905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:14.852603 31913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:14.852983 31905 server_base.cc:1034] running on GCE node
W20260929 18:32:14.852669 31911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:14.852700 31910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:14.853961 31905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:14.855444 31905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:14.856809 31905 hybrid_clock.cc:648] HybridClock initialized: now 1790706734856733 us; error 94 us; skew 500 ppm
Sep 29 18:32:14 dist-test-slave-qcpv krb5kdc[31892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706734, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:14.860438 31905 init.cc:377] Logged in from keytab as kudu/127.28.207.126@KRBTEST.COM (short username kudu)
I20260929 18:32:14.861747 31905 webserver.cc:460] Webserver started at http://127.28.207.126:36601/ using document root <none> and password file <none>
I20260929 18:32:14.862510 31905 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:14.862711 31905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:14.863061 31905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:14.865435 31905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "19d4ca36b3774ed9bd9214a070394142"
format_stamp: "Formatted at 2026-09-29 18:32:14 on dist-test-slave-qcpv"
server_key: "192b3d79a1d571dae8f3e9a009565199"
server_key_iv: "6e6c471f94ad7fbac908f78984d219a3"
server_key_version: "encryptionkey@0"
I20260929 18:32:14.866168 31905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "19d4ca36b3774ed9bd9214a070394142"
format_stamp: "Formatted at 2026-09-29 18:32:14 on dist-test-slave-qcpv"
server_key: "192b3d79a1d571dae8f3e9a009565199"
server_key_iv: "6e6c471f94ad7fbac908f78984d219a3"
server_key_version: "encryptionkey@0"
I20260929 18:32:14.870815 31905 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 18:32:14.873993 31920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:14.875989 31905 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 18:32:14.876134 31905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "19d4ca36b3774ed9bd9214a070394142"
format_stamp: "Formatted at 2026-09-29 18:32:14 on dist-test-slave-qcpv"
server_key: "192b3d79a1d571dae8f3e9a009565199"
server_key_iv: "6e6c471f94ad7fbac908f78984d219a3"
server_key_version: "encryptionkey@0"
I20260929 18:32:14.876252 31905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:14.895738 31905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:14.901552 31905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:14.901960 31905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:14.911134 31905 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:43405
I20260929 18:32:14.911379 31972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:43405 every 8 connection(s)
I20260929 18:32:14.912452 31905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:32:14.916002 31973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:14.916231 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 31905
I20260929 18:32:14.916394 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:32:14.916772 29501 external_mini_cluster.cc:1428] Setting key 330117538bff5bf0c2d9c38a237c7bb3
I20260929 18:32:14.923067 31973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19d4ca36b3774ed9bd9214a070394142: Bootstrap starting.
I20260929 18:32:14.925559 31973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19d4ca36b3774ed9bd9214a070394142: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:14.926378 31973 log.cc:824] T 00000000000000000000000000000000 P 19d4ca36b3774ed9bd9214a070394142: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:14.928792 31973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19d4ca36b3774ed9bd9214a070394142: No bootstrap required, opened a new log
Sep 29 18:32:14 dist-test-slave-qcpv krb5kdc[31892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:32:14.932893 31973 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19d4ca36b3774ed9bd9214a070394142 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d4ca36b3774ed9bd9214a070394142" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43405 } }
I20260929 18:32:14.933223 31973 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19d4ca36b3774ed9bd9214a070394142 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:14.933321 31973 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19d4ca36b3774ed9bd9214a070394142 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19d4ca36b3774ed9bd9214a070394142, State: Initialized, Role: FOLLOWER
I20260929 18:32:14.933853 31973 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19d4ca36b3774ed9bd9214a070394142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d4ca36b3774ed9bd9214a070394142" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43405 } }
I20260929 18:32:14.934013 31973 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19d4ca36b3774ed9bd9214a070394142 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:32:14.934104 31973 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19d4ca36b3774ed9bd9214a070394142 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:32:14.934221 31973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19d4ca36b3774ed9bd9214a070394142 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:14.935333 31973 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19d4ca36b3774ed9bd9214a070394142 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d4ca36b3774ed9bd9214a070394142" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43405 } }
I20260929 18:32:14.935714 31973 leader_election.cc:302] T 00000000000000000000000000000000 P 19d4ca36b3774ed9bd9214a070394142 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19d4ca36b3774ed9bd9214a070394142; no voters:
I20260929 18:32:14.936106 31973 leader_election.cc:288] T 00000000000000000000000000000000 P 19d4ca36b3774ed9bd9214a070394142 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:32:14.936321 31978 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19d4ca36b3774ed9bd9214a070394142 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:32:14.936730 31978 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19d4ca36b3774ed9bd9214a070394142 [term 1 LEADER]: Becoming Leader. State: Replica: 19d4ca36b3774ed9bd9214a070394142, State: Running, Role: LEADER
I20260929 18:32:14.937247 31978 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19d4ca36b3774ed9bd9214a070394142 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d4ca36b3774ed9bd9214a070394142" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43405 } }
I20260929 18:32:14.937624 31973 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19d4ca36b3774ed9bd9214a070394142 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:32:14.939287 31976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:14.918904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42778 (local address 127.28.207.126:43405)
0929 18:32:14.919529 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:14.919551 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:14.919790 (+ 239us) server_negotiation.cc:407] Connection header received
0929 18:32:14.921029 (+ 1239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:14.921059 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:14.921590 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:14.922081 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:14.923213 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:14.924495 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:14.925534 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:14.926923 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:14.927139 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:14.930569 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:14.930576 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:14.930596 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:14.930638 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:14.933925 (+ 3287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:14.934945 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:14.934951 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:14.934957 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:14.935102 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:14.935436 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:14.935440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:14.935442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:14.936127 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:14.936477 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:14.937558 (+ 1081us) server_negotiation.cc:299] Negotiation successful
0929 18:32:14.937980 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":202,"threads_started":1}
I20260929 18:32:14.939435 31980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19d4ca36b3774ed9bd9214a070394142 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19d4ca36b3774ed9bd9214a070394142. Latest consensus state: current_term: 1 leader_uuid: "19d4ca36b3774ed9bd9214a070394142" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d4ca36b3774ed9bd9214a070394142" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43405 } } }
I20260929 18:32:14.939566 31980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19d4ca36b3774ed9bd9214a070394142 [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:14.939965 31979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19d4ca36b3774ed9bd9214a070394142 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19d4ca36b3774ed9bd9214a070394142" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d4ca36b3774ed9bd9214a070394142" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43405 } } }
I20260929 18:32:14.940045 31979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19d4ca36b3774ed9bd9214a070394142 [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:14.940687 31990 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:32:14.944552 31990 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:32:14.952135 31990 catalog_manager.cc:1347] Generated new cluster ID: c42397cbad674faf85292bc64bbb30ff
I20260929 18:32:14.952251 31990 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:32:14.972932 31990 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:32:14.974454 31990 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:32:14.984225 31990 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19d4ca36b3774ed9bd9214a070394142: Generated new TSK 0
I20260929 18:32:14.985332 31990 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.207.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab.
Entry for principal kudu/127.28.207.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab.
I20260929 18:32:15.024385 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.65:0
--local_ip_for_outbound_sockets=127.28.207.65
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:43405
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:42205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:15.171334 31999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:15.171746 31999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:15.171798 31999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:15.176096 31999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:15.176180 31999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:15.176249 31999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.65
I20260929 18:32:15.182250 31999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:42205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:43405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.65
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:15.183713 31999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:15.184958 31999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:32:15.197592 31999 server_base.cc:1034] running on GCE node
W20260929 18:32:15.197754 32004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:15.197589 32007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:15.197570 32005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:15.198609 31999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:15.199541 31999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:15.200762 31999 hybrid_clock.cc:648] HybridClock initialized: now 1790706735200731 us; error 60 us; skew 500 ppm
Sep 29 18:32:15 dist-test-slave-qcpv krb5kdc[31892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706735, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:15.204473 31999 init.cc:377] Logged in from keytab as kudu/127.28.207.65@KRBTEST.COM (short username kudu)
I20260929 18:32:15.205703 31999 webserver.cc:460] Webserver started at http://127.28.207.65:39193/ using document root <none> and password file <none>
I20260929 18:32:15.206434 31999 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:15.206538 31999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:15.206839 31999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:15.209532 31999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-0/data/instance:
uuid: "c7da21d4a98149908460849691886d23"
format_stamp: "Formatted at 2026-09-29 18:32:15 on dist-test-slave-qcpv"
server_key: "ad2241d58010818d5392447a081ae804"
server_key_iv: "426607c37954ece5d8ad8ba7c4e91869"
server_key_version: "encryptionkey@0"
I20260929 18:32:15.210371 31999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance:
uuid: "c7da21d4a98149908460849691886d23"
format_stamp: "Formatted at 2026-09-29 18:32:15 on dist-test-slave-qcpv"
server_key: "ad2241d58010818d5392447a081ae804"
server_key_iv: "426607c37954ece5d8ad8ba7c4e91869"
server_key_version: "encryptionkey@0"
I20260929 18:32:15.215463 31999 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 18:32:15.219084 32014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:15.220981 31999 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:32:15.221257 31999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-0/wal
uuid: "c7da21d4a98149908460849691886d23"
format_stamp: "Formatted at 2026-09-29 18:32:15 on dist-test-slave-qcpv"
server_key: "ad2241d58010818d5392447a081ae804"
server_key_iv: "426607c37954ece5d8ad8ba7c4e91869"
server_key_version: "encryptionkey@0"
I20260929 18:32:15.221426 31999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:15.234681 31999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:15.239204 31999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:15.239439 31999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:15.240192 31999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:32:15.241202 31999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:15.241287 31999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:15.241375 31999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:15.241425 31999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:15.253829 31999 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.65:38677
I20260929 18:32:15.253881 32127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.65:38677 every 8 connection(s)
I20260929 18:32:15.255100 31999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
I20260929 18:32:15.264025 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 31999
I20260929 18:32:15.264210 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance
I20260929 18:32:15.264614 29501 external_mini_cluster.cc:1428] Setting key 87086bffaa3aaba779b86e502230c22e
Sep 29 18:32:15 dist-test-slave-qcpv krb5kdc[31892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706735, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:32:15.272943 31976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:15.257551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:54221 (local address 127.28.207.126:43405)
0929 18:32:15.257757 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:15.257763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:15.259187 (+ 1424us) server_negotiation.cc:407] Connection header received
0929 18:32:15.260592 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:15.260595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:15.260663 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:15.260776 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:15.262206 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:15.262832 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:15.264434 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:15.264812 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:15.264953 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:15.267909 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:15.267917 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:15.267923 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:15.267970 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:15.270165 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:15.270809 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:15.270816 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:15.270821 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:15.270997 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:15.271445 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:15.271451 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:15.271456 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:15.271780 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:15.271902 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:15.272573 (+ 671us) server_negotiation.cc:299] Negotiation successful
0929 18:32:15.272755 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 18:32:15.273837 32130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:15.258182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43405 (local address 127.28.207.65:54221)
0929 18:32:15.258982 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:15.259040 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:15.260318 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:15.261008 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:15.261031 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:15.261476 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:15.262000 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:15.262014 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:15.263035 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:15.263039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:15.264257 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:15.264271 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:15.265037 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:15.265041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:15.265259 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:15.265996 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:15.266013 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:15.267687 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:15.270297 (+ 2610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:15.270303 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:15.270318 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:15.270641 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:15.271161 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:15.271165 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:15.271166 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:15.271324 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:15.271947 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:15.271954 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:15.272243 (+ 289us) client_negotiation.cc:769] Sending connection context
0929 18:32:15.272474 (+ 231us) client_negotiation.cc:240] Negotiation successful
0929 18:32:15.272865 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":161,"threads_started":1}
I20260929 18:32:15.275363 32128 heartbeater.cc:347] Connected to a master server at 127.28.207.126:43405
I20260929 18:32:15.275638 32128 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:15.276189 32128 heartbeater.cc:511] Master 127.28.207.126:43405 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.28.207.66@KRBTEST.COM; defaulting to no policy
I20260929 18:32:15.278343 31937 ts_manager.cc:194] Registered new tserver with Master: c7da21d4a98149908460849691886d23 (127.28.207.65:38677)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.66@KRBTEST.COM" created.
I20260929 18:32:15.280535 31937 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.65@KRBTEST.COM'} at 127.28.207.65:54221
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab.
Entry for principal kudu/127.28.207.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab.
I20260929 18:32:15.299544 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.66:0
--local_ip_for_outbound_sockets=127.28.207.66
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:43405
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:42205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:15.431043 32133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:15.431365 32133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:15.431412 32133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:15.436196 32133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:15.436297 32133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:15.436368 32133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.66
I20260929 18:32:15.442541 32133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:42205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:43405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.66
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:15.443789 32133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:15.445026 32133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:15.456837 32139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:15.456892 32141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:15.456892 32138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:15.457355 32133 server_base.cc:1034] running on GCE node
I20260929 18:32:15.457945 32133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:15.458846 32133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:15.460247 32133 hybrid_clock.cc:648] HybridClock initialized: now 1790706735460201 us; error 78 us; skew 500 ppm
Sep 29 18:32:15 dist-test-slave-qcpv krb5kdc[31892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706735, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:15.464514 32133 init.cc:377] Logged in from keytab as kudu/127.28.207.66@KRBTEST.COM (short username kudu)
I20260929 18:32:15.465648 32133 webserver.cc:460] Webserver started at http://127.28.207.66:39121/ using document root <none> and password file <none>
I20260929 18:32:15.466289 32133 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:15.466357 32133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:15.466591 32133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:15.469009 32133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-1/data/instance:
uuid: "fab5675611ee4acf8d99640013b9b3ed"
format_stamp: "Formatted at 2026-09-29 18:32:15 on dist-test-slave-qcpv"
server_key: "74ed1be36bb6fdcdaabbdc65c64ee9d5"
server_key_iv: "65059c8eb531deb6705b932e72120123"
server_key_version: "encryptionkey@0"
I20260929 18:32:15.469630 32133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance:
uuid: "fab5675611ee4acf8d99640013b9b3ed"
format_stamp: "Formatted at 2026-09-29 18:32:15 on dist-test-slave-qcpv"
server_key: "74ed1be36bb6fdcdaabbdc65c64ee9d5"
server_key_iv: "65059c8eb531deb6705b932e72120123"
server_key_version: "encryptionkey@0"
I20260929 18:32:15.473927 32133 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:32:15.476855 32148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:15.478540 32133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:32:15.478698 32133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-1/wal
uuid: "fab5675611ee4acf8d99640013b9b3ed"
format_stamp: "Formatted at 2026-09-29 18:32:15 on dist-test-slave-qcpv"
server_key: "74ed1be36bb6fdcdaabbdc65c64ee9d5"
server_key_iv: "65059c8eb531deb6705b932e72120123"
server_key_version: "encryptionkey@0"
I20260929 18:32:15.478826 32133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:15.494484 32133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:15.498653 32133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:15.498966 32133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:15.499775 32133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:32:15.500901 32133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:15.500993 32133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:15.501086 32133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:15.501140 32133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:15.513880 32133 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.66:36763
I20260929 18:32:15.513967 32261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.66:36763 every 8 connection(s)
I20260929 18:32:15.514987 32133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
I20260929 18:32:15.518448 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 32133
I20260929 18:32:15.518599 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance
I20260929 18:32:15.519029 29501 external_mini_cluster.cc:1428] Setting key 5ec731c9419cd7e78091f64fec64c3ff
Sep 29 18:32:15 dist-test-slave-qcpv krb5kdc[31892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706735, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.207.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.67@KRBTEST.COM" created.
I20260929 18:32:15.535758 31976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:15.517675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:40667 (local address 127.28.207.126:43405)
0929 18:32:15.517904 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:15.517911 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:15.520022 (+ 2111us) server_negotiation.cc:407] Connection header received
0929 18:32:15.521378 (+ 1356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:15.521381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:15.521440 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:15.521533 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:15.524121 (+ 2588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:15.524762 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:15.527467 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:15.527751 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:15.527881 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:15.531697 (+ 3816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:15.531700 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:15.531702 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:15.531732 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:15.533348 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:15.533922 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:15.533930 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:15.533931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:15.534032 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:15.534432 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:15.534435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:15.534437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:15.534643 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:15.534783 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:15.535444 (+ 661us) server_negotiation.cc:299] Negotiation successful
0929 18:32:15.535580 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 18:32:15.536659 32264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:15.518108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43405 (local address 127.28.207.66:40667)
0929 18:32:15.519493 (+ 1385us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:15.519792 (+ 299us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:15.521102 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:15.521761 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:15.521769 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:15.522380 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:15.523864 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:15.523900 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:15.525065 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:15.525073 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:15.527266 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:15.527284 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:15.527990 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:15.527998 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:15.528291 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:15.529214 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:15.529246 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:15.531455 (+ 2209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:15.533482 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:15.533489 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:15.533508 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:15.533804 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:15.534114 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:15.534118 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:15.534121 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:15.534314 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:15.534789 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:15.534798 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:15.535186 (+ 388us) client_negotiation.cc:769] Sending connection context
0929 18:32:15.535392 (+ 206us) client_negotiation.cc:240] Negotiation successful
0929 18:32:15.535767 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":931,"thread_start_us":203,"threads_started":1}
I20260929 18:32:15.538246 32262 heartbeater.cc:347] Connected to a master server at 127.28.207.126:43405
I20260929 18:32:15.538578 32262 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:15.539187 32262 heartbeater.cc:511] Master 127.28.207.126:43405 requested a full tablet report, sending...
I20260929 18:32:15.540494 31937 ts_manager.cc:194] Registered new tserver with Master: fab5675611ee4acf8d99640013b9b3ed (127.28.207.66:36763)
I20260929 18:32:15.541522 31937 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.66@KRBTEST.COM'} at 127.28.207.66:40667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab.
Entry for principal kudu/127.28.207.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab.
I20260929 18:32:15.555819 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.67:0
--local_ip_for_outbound_sockets=127.28.207.67
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:43405
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:42205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:15.697002 32267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:15.697585 32267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:15.697645 32267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:15.703528 32267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:15.703652 32267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:15.703727 32267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.67
I20260929 18:32:15.710325 32267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:42205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:43405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.67
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:15.711920 32267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:15.713248 32267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:15.727608 32273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:15.727595 32272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:15.727595 32275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:15.729137 32267 server_base.cc:1034] running on GCE node
I20260929 18:32:15.730141 32267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:15.731294 32267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:15.732743 32267 hybrid_clock.cc:648] HybridClock initialized: now 1790706735732711 us; error 59 us; skew 500 ppm
Sep 29 18:32:15 dist-test-slave-qcpv krb5kdc[31892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706735, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:15.736358 32267 init.cc:377] Logged in from keytab as kudu/127.28.207.67@KRBTEST.COM (short username kudu)
I20260929 18:32:15.737466 32267 webserver.cc:460] Webserver started at http://127.28.207.67:45619/ using document root <none> and password file <none>
I20260929 18:32:15.738180 32267 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:15.738253 32267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:15.738546 32267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:15.741166 32267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-2/data/instance:
uuid: "30011079166e44ea95691e6a8014b854"
format_stamp: "Formatted at 2026-09-29 18:32:15 on dist-test-slave-qcpv"
server_key: "83b28442bdda6691cb6b6e96d1214355"
server_key_iv: "7c287c5a7f2ad7523454f35763a50cc5"
server_key_version: "encryptionkey@0"
I20260929 18:32:15.741772 32267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance:
uuid: "30011079166e44ea95691e6a8014b854"
format_stamp: "Formatted at 2026-09-29 18:32:15 on dist-test-slave-qcpv"
server_key: "83b28442bdda6691cb6b6e96d1214355"
server_key_iv: "7c287c5a7f2ad7523454f35763a50cc5"
server_key_version: "encryptionkey@0"
I20260929 18:32:15.746243 32267 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:32:15.749258 32282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:15.751062 32267 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:32:15.751185 32267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-2/wal
uuid: "30011079166e44ea95691e6a8014b854"
format_stamp: "Formatted at 2026-09-29 18:32:15 on dist-test-slave-qcpv"
server_key: "83b28442bdda6691cb6b6e96d1214355"
server_key_iv: "7c287c5a7f2ad7523454f35763a50cc5"
server_key_version: "encryptionkey@0"
I20260929 18:32:15.751333 32267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:15.779018 32267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:15.783273 32267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:15.783502 32267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:15.784152 32267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:32:15.785050 32267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:15.785102 32267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:15.785171 32267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:15.785213 32267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:15.796269 32267 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.67:36387
I20260929 18:32:15.796411 32395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.67:36387 every 8 connection(s)
I20260929 18:32:15.797317 32267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
I20260929 18:32:15.805555 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 32267
I20260929 18:32:15.805666 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance
I20260929 18:32:15.805958 29501 external_mini_cluster.cc:1428] Setting key a998ae6897f04cbbe14144bcfb0b697f
Sep 29 18:32:15 dist-test-slave-qcpv krb5kdc[31892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706735, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:32:15.813252 31976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:15.799837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:48539 (local address 127.28.207.126:43405)
0929 18:32:15.800036 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:15.800040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:15.801200 (+ 1160us) server_negotiation.cc:407] Connection header received
0929 18:32:15.802261 (+ 1061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:15.802265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:15.802325 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:15.802426 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:15.803983 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:15.804541 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:15.806211 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:15.806475 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:15.806608 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:15.809485 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:15.809491 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:15.809499 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:15.809523 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:15.810999 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:15.811510 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:15.811516 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:15.811517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:15.811597 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:15.812067 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:15.812072 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:15.812074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:15.812300 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:15.812415 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:15.812940 (+ 525us) server_negotiation.cc:299] Negotiation successful
0929 18:32:15.813070 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 18:32:15.814131 32398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:15.800155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43405 (local address 127.28.207.67:48539)
0929 18:32:15.800967 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:15.801022 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:15.802010 (+ 988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:15.802682 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:15.802691 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:15.803181 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:15.803819 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:15.803835 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:15.804702 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:15.804706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:15.806072 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:15.806084 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:15.806711 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:15.806719 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:15.806992 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:15.807653 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:15.807670 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:15.809297 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:15.811120 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:15.811126 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:15.811143 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:15.811403 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:15.811798 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:15.811802 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:15.811804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:15.811958 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:15.812414 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:15.812422 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:15.812698 (+ 276us) client_negotiation.cc:769] Sending connection context
0929 18:32:15.812900 (+ 202us) client_negotiation.cc:240] Negotiation successful
0929 18:32:15.813249 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":116,"threads_started":1}
I20260929 18:32:15.815696 32396 heartbeater.cc:347] Connected to a master server at 127.28.207.126:43405
I20260929 18:32:15.816013 32396 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:15.816570 32396 heartbeater.cc:511] Master 127.28.207.126:43405 requested a full tablet report, sending...
I20260929 18:32:15.817788 31937 ts_manager.cc:194] Registered new tserver with Master: 30011079166e44ea95691e6a8014b854 (127.28.207.67:36387)
I20260929 18:32:15.818411 31937 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.67@KRBTEST.COM'} at 127.28.207.67:48539
I20260929 18:32:15.820982 29501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 18:32:15.838999 31976 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:15.836665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39096 (local address 127.28.207.126:43405)
0929 18:32:15.837043 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:15.837050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:15.837222 (+ 172us) server_negotiation.cc:407] Connection header received
0929 18:32:15.837453 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:15.837456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:15.837510 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:15.837608 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:15.838726 (+ 1118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39096: BlockingRecv error: recv got EOF from 127.0.0.1:39096 (error 108)
Metrics: {"server-negotiator.queue_time_us":142}
I20260929 18:32:15.841010 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 31999
I20260929 18:32:15.847872 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 32133
I20260929 18:32:15.854758 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 32267
I20260929 18:32:15.860687 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 31905
2026-09-29T18:32:15Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2444 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:32:15 dist-test-slave-qcpv krb5kdc[32408](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:32:15 dist-test-slave-qcpv krb5kdc[32408](info): set up 2 sockets
Sep 29 18:32:15 dist-test-slave-qcpv krb5kdc[32408](info): commencing operation
krb5kdc: starting...
W20260929 18:32:16.927487 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.007s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:32:17 dist-test-slave-qcpv krb5kdc[32408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:32:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:32:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.207.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
I20260929 18:32:17.076213 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:38621
--webserver_interface=127.28.207.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:37201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:38621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:17.215343 32421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:17.215732 32421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:17.215780 32421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:17.220652 32421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:32:17.220760 32421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:17.220783 32421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:17.220803 32421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:32:17.220824 32421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:32:17.227995 32421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:37201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:38621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:38621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:17.229679 32421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:17.231765 32421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:17.243681 32427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:17.243736 32429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:17.243625 32421 server_base.cc:1034] running on GCE node
W20260929 18:32:17.243664 32426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:17.244947 32421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:17.246537 32421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:17.247785 32421 hybrid_clock.cc:648] HybridClock initialized: now 1790706737247789 us; error 80 us; skew 500 ppm
Sep 29 18:32:17 dist-test-slave-qcpv krb5kdc[32408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706737, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:17.251780 32421 init.cc:377] Logged in from keytab as kudu/127.28.207.126@KRBTEST.COM (short username kudu)
I20260929 18:32:17.253391 32421 webserver.cc:460] Webserver started at http://127.28.207.126:40207/ using document root <none> and password file <none>
I20260929 18:32:17.254169 32421 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:17.254279 32421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:17.254570 32421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:17.257339 32421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "b63ac8810b2540ada018ddc0400cc426"
format_stamp: "Formatted at 2026-09-29 18:32:17 on dist-test-slave-qcpv"
server_key: "67edc4eefd9a393ce1873f4cda73c6d8"
server_key_iv: "ef005c74bd1a420932b8fdbf2629b769"
server_key_version: "encryptionkey@0"
I20260929 18:32:17.258036 32421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "b63ac8810b2540ada018ddc0400cc426"
format_stamp: "Formatted at 2026-09-29 18:32:17 on dist-test-slave-qcpv"
server_key: "67edc4eefd9a393ce1873f4cda73c6d8"
server_key_iv: "ef005c74bd1a420932b8fdbf2629b769"
server_key_version: "encryptionkey@0"
I20260929 18:32:17.263711 32421 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260929 18:32:17.266786 32436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:17.268597 32421 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 18:32:17.268744 32421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "b63ac8810b2540ada018ddc0400cc426"
format_stamp: "Formatted at 2026-09-29 18:32:17 on dist-test-slave-qcpv"
server_key: "67edc4eefd9a393ce1873f4cda73c6d8"
server_key_iv: "ef005c74bd1a420932b8fdbf2629b769"
server_key_version: "encryptionkey@0"
I20260929 18:32:17.268922 32421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:17.284765 32421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:17.288313 32421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:17.288547 32421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:17.296788 32421 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:38621
I20260929 18:32:17.297153 32488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:38621 every 8 connection(s)
I20260929 18:32:17.297966 32421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:32:17.301908 32489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:17.305023 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 32421
I20260929 18:32:17.305197 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:32:17.305496 29501 external_mini_cluster.cc:1428] Setting key 4dc7eec4d7b01316cbad1566f059ecf2
I20260929 18:32:17.309801 32489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b63ac8810b2540ada018ddc0400cc426: Bootstrap starting.
I20260929 18:32:17.313354 32489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b63ac8810b2540ada018ddc0400cc426: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:17.314467 32489 log.cc:824] T 00000000000000000000000000000000 P b63ac8810b2540ada018ddc0400cc426: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:32:17 dist-test-slave-qcpv krb5kdc[32408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:32:17.317713 32489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b63ac8810b2540ada018ddc0400cc426: No bootstrap required, opened a new log
I20260929 18:32:17.324113 32489 raft_consensus.cc:348] T 00000000000000000000000000000000 P b63ac8810b2540ada018ddc0400cc426 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b63ac8810b2540ada018ddc0400cc426" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 38621 } }
I20260929 18:32:17.324509 32489 raft_consensus.cc:374] T 00000000000000000000000000000000 P b63ac8810b2540ada018ddc0400cc426 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:17.324579 32489 raft_consensus.cc:729] T 00000000000000000000000000000000 P b63ac8810b2540ada018ddc0400cc426 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b63ac8810b2540ada018ddc0400cc426, State: Initialized, Role: FOLLOWER
I20260929 18:32:17.325217 32489 consensus_queue.cc:260] T 00000000000000000000000000000000 P b63ac8810b2540ada018ddc0400cc426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b63ac8810b2540ada018ddc0400cc426" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 38621 } }
I20260929 18:32:17.325380 32492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:17.307261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58880 (local address 127.28.207.126:38621)
0929 18:32:17.307943 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:17.307964 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:17.308023 (+ 59us) server_negotiation.cc:407] Connection header received
0929 18:32:17.308951 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:17.308977 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:17.309417 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:17.309873 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:17.310820 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:17.312093 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:17.313110 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:17.313799 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:17.313998 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:17.317529 (+ 3531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:17.317537 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:17.317559 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:17.317598 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:17.320496 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:17.321225 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:17.321236 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:17.321245 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:17.321463 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:17.321824 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:17.321827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:17.321829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:17.322522 (+ 693us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:17.322768 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:17.323293 (+ 525us) server_negotiation.cc:299] Negotiation successful
0929 18:32:17.323984 (+ 691us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":195,"threads_started":1}
I20260929 18:32:17.325407 32489 raft_consensus.cc:388] T 00000000000000000000000000000000 P b63ac8810b2540ada018ddc0400cc426 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:32:17.325476 32489 raft_consensus.cc:482] T 00000000000000000000000000000000 P b63ac8810b2540ada018ddc0400cc426 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:32:17.325579 32489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b63ac8810b2540ada018ddc0400cc426 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:17.327064 32489 raft_consensus.cc:504] T 00000000000000000000000000000000 P b63ac8810b2540ada018ddc0400cc426 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b63ac8810b2540ada018ddc0400cc426" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 38621 } }
I20260929 18:32:17.327538 32489 leader_election.cc:302] T 00000000000000000000000000000000 P b63ac8810b2540ada018ddc0400cc426 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b63ac8810b2540ada018ddc0400cc426; no voters:
I20260929 18:32:17.328019 32489 leader_election.cc:288] T 00000000000000000000000000000000 P b63ac8810b2540ada018ddc0400cc426 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:32:17.328258 32494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b63ac8810b2540ada018ddc0400cc426 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:32:17.328716 32494 raft_consensus.cc:686] T 00000000000000000000000000000000 P b63ac8810b2540ada018ddc0400cc426 [term 1 LEADER]: Becoming Leader. State: Replica: b63ac8810b2540ada018ddc0400cc426, State: Running, Role: LEADER
I20260929 18:32:17.329210 32494 consensus_queue.cc:237] T 00000000000000000000000000000000 P b63ac8810b2540ada018ddc0400cc426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b63ac8810b2540ada018ddc0400cc426" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 38621 } }
I20260929 18:32:17.329429 29501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:32:17.329788 32489 sys_catalog.cc:564] T 00000000000000000000000000000000 P b63ac8810b2540ada018ddc0400cc426 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:32:17.331564 32495 sys_catalog.cc:455] T 00000000000000000000000000000000 P b63ac8810b2540ada018ddc0400cc426 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b63ac8810b2540ada018ddc0400cc426" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b63ac8810b2540ada018ddc0400cc426" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 38621 } } }
I20260929 18:32:17.331756 32495 sys_catalog.cc:458] T 00000000000000000000000000000000 P b63ac8810b2540ada018ddc0400cc426 [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:17.331564 32496 sys_catalog.cc:455] T 00000000000000000000000000000000 P b63ac8810b2540ada018ddc0400cc426 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b63ac8810b2540ada018ddc0400cc426. Latest consensus state: current_term: 1 leader_uuid: "b63ac8810b2540ada018ddc0400cc426" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b63ac8810b2540ada018ddc0400cc426" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 38621 } } }
I20260929 18:32:17.331841 32496 sys_catalog.cc:458] T 00000000000000000000000000000000 P b63ac8810b2540ada018ddc0400cc426 [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:17.332211 32503 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:32:17.335743 32503 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:32:17.344157 32503 catalog_manager.cc:1347] Generated new cluster ID: 834f8637a22f4fd4af3a9b6909594bbb
I20260929 18:32:17.344245 32503 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:32:17.355646 32492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:17.349630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58894 (local address 127.28.207.126:38621)
0929 18:32:17.349786 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:17.349791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:17.350080 (+ 289us) server_negotiation.cc:407] Connection header received
0929 18:32:17.350282 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:17.350285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:17.350334 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:17.350427 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:17.351081 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:17.351673 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:17.352598 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:17.352883 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:17.353001 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:17.354004 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:17.354008 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:17.354013 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:32:17.354994 (+ 981us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:17.355035 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:32:17.355056 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:17.355183 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:17.355325 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 18:32:17.355420 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 18:32:17.366494 32503 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:32:17.367925 32503 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:32:17.373208 32503 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b63ac8810b2540ada018ddc0400cc426: Generated new TSK 0
I20260929 18:32:17.374089 32503 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:32:17.383730 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 32421
2026-09-29T18:32:17Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1521 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:32:17 dist-test-slave-qcpv krb5kdc[32522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:32:17 dist-test-slave-qcpv krb5kdc[32522](info): set up 2 sockets
Sep 29 18:32:17 dist-test-slave-qcpv krb5kdc[32522](info): commencing operation
krb5kdc: starting...
W20260929 18:32:18.487506 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:32:18 dist-test-slave-qcpv krb5kdc[32522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:32:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:32:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.207.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
I20260929 18:32:18.630955 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:37315
--webserver_interface=127.28.207.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:43939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:37315
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:18.753248 32535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:18.753536 32535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:18.753578 32535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:18.757784 32535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:32:18.757862 32535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:18.757882 32535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:18.757898 32535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:32:18.757915 32535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:32:18.763463 32535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:43939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:37315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:37315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:18.764601 32535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:18.765640 32535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:18.775362 32541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:18.775364 32540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:18.775365 32543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:18.775700 32535 server_base.cc:1034] running on GCE node
I20260929 18:32:18.776612 32535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:18.778101 32535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:18.781503 32535 hybrid_clock.cc:648] HybridClock initialized: now 1790706738781442 us; error 69 us; skew 500 ppm
Sep 29 18:32:18 dist-test-slave-qcpv krb5kdc[32522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706738, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:18.792651 32535 init.cc:377] Logged in from keytab as kudu/127.28.207.126@KRBTEST.COM (short username kudu)
I20260929 18:32:18.794020 32535 webserver.cc:460] Webserver started at http://127.28.207.126:32975/ using document root <none> and password file <none>
I20260929 18:32:18.794747 32535 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:18.794817 32535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:18.795126 32535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:18.798000 32535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "f6f159e2a9314e79878fd0f947ac7daa"
format_stamp: "Formatted at 2026-09-29 18:32:18 on dist-test-slave-qcpv"
server_key: "8302ed71c892324251771f24fc6ea217"
server_key_iv: "720a3536876a61e717f9dd6adb1086de"
server_key_version: "encryptionkey@0"
I20260929 18:32:18.798704 32535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "f6f159e2a9314e79878fd0f947ac7daa"
format_stamp: "Formatted at 2026-09-29 18:32:18 on dist-test-slave-qcpv"
server_key: "8302ed71c892324251771f24fc6ea217"
server_key_iv: "720a3536876a61e717f9dd6adb1086de"
server_key_version: "encryptionkey@0"
I20260929 18:32:18.803930 32535 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 18:32:18.807391 32550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:18.809649 32535 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260929 18:32:18.809897 32535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "f6f159e2a9314e79878fd0f947ac7daa"
format_stamp: "Formatted at 2026-09-29 18:32:18 on dist-test-slave-qcpv"
server_key: "8302ed71c892324251771f24fc6ea217"
server_key_iv: "720a3536876a61e717f9dd6adb1086de"
server_key_version: "encryptionkey@0"
I20260929 18:32:18.810065 32535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:18.834914 32535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:18.839128 32535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:18.839331 32535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:18.849522 32535 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:37315
I20260929 18:32:18.850057 32602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:37315 every 8 connection(s)
I20260929 18:32:18.851310 32535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:32:18.854705 32603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:18.860436 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 32535
I20260929 18:32:18.860608 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:32:18.860913 29501 external_mini_cluster.cc:1428] Setting key a928c75be2b818687b5d350ed644883d
I20260929 18:32:18.861838 32603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6f159e2a9314e79878fd0f947ac7daa: Bootstrap starting.
I20260929 18:32:18.865070 32603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6f159e2a9314e79878fd0f947ac7daa: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:18.865976 32603 log.cc:824] T 00000000000000000000000000000000 P f6f159e2a9314e79878fd0f947ac7daa: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:18.869149 32603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6f159e2a9314e79878fd0f947ac7daa: No bootstrap required, opened a new log
Sep 29 18:32:18 dist-test-slave-qcpv krb5kdc[32522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:32:18.874058 32603 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6f159e2a9314e79878fd0f947ac7daa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f159e2a9314e79878fd0f947ac7daa" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 37315 } }
I20260929 18:32:18.874534 32603 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6f159e2a9314e79878fd0f947ac7daa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:18.874631 32603 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6f159e2a9314e79878fd0f947ac7daa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6f159e2a9314e79878fd0f947ac7daa, State: Initialized, Role: FOLLOWER
I20260929 18:32:18.875388 32603 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6f159e2a9314e79878fd0f947ac7daa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f159e2a9314e79878fd0f947ac7daa" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 37315 } }
I20260929 18:32:18.875566 32603 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6f159e2a9314e79878fd0f947ac7daa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:32:18.875614 32603 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6f159e2a9314e79878fd0f947ac7daa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:32:18.875777 32603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6f159e2a9314e79878fd0f947ac7daa [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:18.877102 32603 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6f159e2a9314e79878fd0f947ac7daa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f159e2a9314e79878fd0f947ac7daa" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 37315 } }
I20260929 18:32:18.877542 32603 leader_election.cc:302] T 00000000000000000000000000000000 P f6f159e2a9314e79878fd0f947ac7daa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6f159e2a9314e79878fd0f947ac7daa; no voters:
I20260929 18:32:18.877924 32603 leader_election.cc:288] T 00000000000000000000000000000000 P f6f159e2a9314e79878fd0f947ac7daa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:32:18.878487 32608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6f159e2a9314e79878fd0f947ac7daa [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:32:18.878839 32608 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6f159e2a9314e79878fd0f947ac7daa [term 1 LEADER]: Becoming Leader. State: Replica: f6f159e2a9314e79878fd0f947ac7daa, State: Running, Role: LEADER
I20260929 18:32:18.879271 32608 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6f159e2a9314e79878fd0f947ac7daa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f159e2a9314e79878fd0f947ac7daa" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 37315 } }
I20260929 18:32:18.879670 32603 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6f159e2a9314e79878fd0f947ac7daa [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:32:18.881203 32609 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6f159e2a9314e79878fd0f947ac7daa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6f159e2a9314e79878fd0f947ac7daa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f159e2a9314e79878fd0f947ac7daa" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 37315 } } }
I20260929 18:32:18.881361 32609 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6f159e2a9314e79878fd0f947ac7daa [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:18.881809 32610 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6f159e2a9314e79878fd0f947ac7daa [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6f159e2a9314e79878fd0f947ac7daa. Latest consensus state: current_term: 1 leader_uuid: "f6f159e2a9314e79878fd0f947ac7daa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6f159e2a9314e79878fd0f947ac7daa" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 37315 } } }
I20260929 18:32:18.881944 32610 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6f159e2a9314e79878fd0f947ac7daa [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:18.882266 32619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:32:18.882817 32606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:18.863007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42710 (local address 127.28.207.126:37315)
0929 18:32:18.863600 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:18.863620 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:18.863671 (+ 51us) server_negotiation.cc:407] Connection header received
0929 18:32:18.864819 (+ 1148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:18.864846 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:18.865435 (+ 589us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:18.865872 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:18.867203 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:18.868414 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:18.869758 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:18.870409 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:18.870615 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:18.873967 (+ 3352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:18.873974 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:18.873992 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:18.874036 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:18.877048 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:18.877962 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:18.877971 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:18.877979 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:18.878281 (+ 302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:18.878780 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:18.878784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:18.878786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:18.879415 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:18.879638 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:18.880633 (+ 995us) server_negotiation.cc:299] Negotiation successful
0929 18:32:18.881345 (+ 712us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":195,"threads_started":1}
I20260929 18:32:18.887130 32619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:32:18.894070 32619 catalog_manager.cc:1347] Generated new cluster ID: 0adfc38ea4464658ab8305a59c6259ed
I20260929 18:32:18.894369 32619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:32:18.910358 32619 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:32:18.911676 32619 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:32:18.918375 32619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6f159e2a9314e79878fd0f947ac7daa: Generated new TSK 0
I20260929 18:32:18.919286 32619 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.207.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab.
Entry for principal kudu/127.28.207.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab.
I20260929 18:32:18.959664 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.65:0
--local_ip_for_outbound_sockets=127.28.207.65
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:37315
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:43939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:19.115769 32629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:19.116160 32629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:19.116241 32629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:19.121236 32629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:19.121377 32629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:19.121497 32629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.65
I20260929 18:32:19.128458 32629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:43939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:37315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.65
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:19.129721 32629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:19.131207 32629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:19.143692 32634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:19.143692 32635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:19.143839 32637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:19.144104 32629 server_base.cc:1034] running on GCE node
I20260929 18:32:19.144870 32629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:19.145802 32629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:19.147137 32629 hybrid_clock.cc:648] HybridClock initialized: now 1790706739147078 us; error 85 us; skew 500 ppm
Sep 29 18:32:19 dist-test-slave-qcpv krb5kdc[32522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706739, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:19.151137 32629 init.cc:377] Logged in from keytab as kudu/127.28.207.65@KRBTEST.COM (short username kudu)
I20260929 18:32:19.152441 32629 webserver.cc:460] Webserver started at http://127.28.207.65:43901/ using document root <none> and password file <none>
I20260929 18:32:19.153131 32629 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:19.153203 32629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:19.153414 32629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:19.156087 32629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-0/data/instance:
uuid: "892b716a61bb463e92e7fa5aaa2c91dc"
format_stamp: "Formatted at 2026-09-29 18:32:19 on dist-test-slave-qcpv"
server_key: "3ecd6481f6e1b5eebf5fafac2bea405a"
server_key_iv: "4e096eae68b24c2c77d9423ccb7f170e"
server_key_version: "encryptionkey@0"
I20260929 18:32:19.156719 32629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance:
uuid: "892b716a61bb463e92e7fa5aaa2c91dc"
format_stamp: "Formatted at 2026-09-29 18:32:19 on dist-test-slave-qcpv"
server_key: "3ecd6481f6e1b5eebf5fafac2bea405a"
server_key_iv: "4e096eae68b24c2c77d9423ccb7f170e"
server_key_version: "encryptionkey@0"
I20260929 18:32:19.161682 32629 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 18:32:19.165141 32644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:19.167285 32629 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:32:19.167449 32629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-0/wal
uuid: "892b716a61bb463e92e7fa5aaa2c91dc"
format_stamp: "Formatted at 2026-09-29 18:32:19 on dist-test-slave-qcpv"
server_key: "3ecd6481f6e1b5eebf5fafac2bea405a"
server_key_iv: "4e096eae68b24c2c77d9423ccb7f170e"
server_key_version: "encryptionkey@0"
I20260929 18:32:19.167585 32629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:19.202473 32629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:19.206614 32629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:19.206820 32629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:19.207571 32629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:32:19.208541 32629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:19.208592 32629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:19.208639 32629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:19.208655 32629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:19.220487 32629 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.65:44861
I20260929 18:32:19.220554 32757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.65:44861 every 8 connection(s)
I20260929 18:32:19.221720 32629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
I20260929 18:32:19.230193 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 32629
I20260929 18:32:19.230317 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance
I20260929 18:32:19.230695 29501 external_mini_cluster.cc:1428] Setting key 14e74eabdccb9fc49575858601c06a70
Sep 29 18:32:19 dist-test-slave-qcpv krb5kdc[32522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706739, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:32:19.239652 32606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:19.224078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:54355 (local address 127.28.207.126:37315)
0929 18:32:19.224313 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:19.224317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:19.225398 (+ 1081us) server_negotiation.cc:407] Connection header received
0929 18:32:19.226903 (+ 1505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:19.226907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:19.226973 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:19.227101 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:19.228381 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:19.228942 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:19.230127 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:19.230469 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:19.230710 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:19.234273 (+ 3563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:19.234282 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:19.234289 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:19.234339 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:19.236407 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:19.237158 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:19.237162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:19.237163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:19.237263 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:19.237896 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:19.237900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:19.237902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:19.238175 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:19.238274 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:19.239251 (+ 977us) server_negotiation.cc:299] Negotiation successful
0929 18:32:19.239417 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260929 18:32:19.240687 32760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:19.224409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:37315 (local address 127.28.207.65:54355)
0929 18:32:19.225196 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:19.225252 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:19.226638 (+ 1386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:19.227294 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:19.227301 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:19.227697 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:19.228228 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:19.228242 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:19.229078 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:19.229084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:19.229955 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:19.229964 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:19.230734 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:19.230740 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:19.230978 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:19.231731 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:19.231749 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:19.233976 (+ 2227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:19.236575 (+ 2599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:19.236585 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:19.236605 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:19.237008 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:19.237385 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:19.237391 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:19.237396 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:19.237608 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:19.238314 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:19.238325 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:19.238740 (+ 415us) client_negotiation.cc:769] Sending connection context
0929 18:32:19.239026 (+ 286us) client_negotiation.cc:240] Negotiation successful
0929 18:32:19.239467 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":148,"threads_started":1}
WARNING: no policy specified for kudu/127.28.207.66@KRBTEST.COM; defaulting to no policy
I20260929 18:32:19.242506 32758 heartbeater.cc:347] Connected to a master server at 127.28.207.126:37315
I20260929 18:32:19.242795 32758 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:19.243389 32758 heartbeater.cc:511] Master 127.28.207.126:37315 requested a full tablet report, sending...
I20260929 18:32:19.245201 32567 ts_manager.cc:194] Registered new tserver with Master: 892b716a61bb463e92e7fa5aaa2c91dc (127.28.207.65:44861)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.66@KRBTEST.COM" created.
I20260929 18:32:19.246563 32567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.65@KRBTEST.COM'} at 127.28.207.65:54355
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab.
Entry for principal kudu/127.28.207.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab.
I20260929 18:32:19.267048 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.66:0
--local_ip_for_outbound_sockets=127.28.207.66
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:37315
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:43939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:19.388183 32763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:19.388465 32763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:19.388512 32763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:19.392608 32763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:19.392678 32763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:19.392736 32763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.66
I20260929 18:32:19.398082 32763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:43939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:37315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.66
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:19.399220 32763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:19.400243 32763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:19.410892 301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:19.410970 300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:19.411145 32763 server_base.cc:1034] running on GCE node
W20260929 18:32:19.411387 303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:19.411835 32763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:19.412483 32763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:19.415845 32763 hybrid_clock.cc:648] HybridClock initialized: now 1790706739415753 us; error 120 us; skew 500 ppm
Sep 29 18:32:19 dist-test-slave-qcpv krb5kdc[32522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706739, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:19.419514 32763 init.cc:377] Logged in from keytab as kudu/127.28.207.66@KRBTEST.COM (short username kudu)
I20260929 18:32:19.420512 32763 webserver.cc:460] Webserver started at http://127.28.207.66:41087/ using document root <none> and password file <none>
I20260929 18:32:19.421119 32763 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:19.421211 32763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:19.421450 32763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:19.423667 32763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-1/data/instance:
uuid: "0de0afff64a44deabf1b83e270b42bff"
format_stamp: "Formatted at 2026-09-29 18:32:19 on dist-test-slave-qcpv"
server_key: "4152c9260c97e24e81b9e4c7d340b1c2"
server_key_iv: "b528050a8105d844e9f5061e2edff339"
server_key_version: "encryptionkey@0"
I20260929 18:32:19.424263 32763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance:
uuid: "0de0afff64a44deabf1b83e270b42bff"
format_stamp: "Formatted at 2026-09-29 18:32:19 on dist-test-slave-qcpv"
server_key: "4152c9260c97e24e81b9e4c7d340b1c2"
server_key_iv: "b528050a8105d844e9f5061e2edff339"
server_key_version: "encryptionkey@0"
I20260929 18:32:19.428424 32763 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260929 18:32:19.431402 310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:19.433127 32763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:32:19.433308 32763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-1/wal
uuid: "0de0afff64a44deabf1b83e270b42bff"
format_stamp: "Formatted at 2026-09-29 18:32:19 on dist-test-slave-qcpv"
server_key: "4152c9260c97e24e81b9e4c7d340b1c2"
server_key_iv: "b528050a8105d844e9f5061e2edff339"
server_key_version: "encryptionkey@0"
I20260929 18:32:19.433440 32763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:19.451258 32763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:19.454748 32763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:19.454994 32763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:19.455611 32763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:32:19.456508 32763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:19.456597 32763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:19.456672 32763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:19.456717 32763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:19.467815 32763 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.66:33785
I20260929 18:32:19.467839 423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.66:33785 every 8 connection(s)
I20260929 18:32:19.468819 32763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
I20260929 18:32:19.476464 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 32763
I20260929 18:32:19.476639 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance
I20260929 18:32:19.476966 29501 external_mini_cluster.cc:1428] Setting key 6b78e30c26bdc864ab93ceedf96a9be8
Sep 29 18:32:19 dist-test-slave-qcpv krb5kdc[32522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706739, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:32:19.487304 32606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:19.470830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:55659 (local address 127.28.207.126:37315)
0929 18:32:19.471059 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:19.471063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:19.472160 (+ 1097us) server_negotiation.cc:407] Connection header received
0929 18:32:19.474013 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:19.474017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:19.474095 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:19.474218 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:19.475779 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:19.476530 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:19.478384 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:19.478740 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:19.478906 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:19.482713 (+ 3807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:19.482718 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:19.482721 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:19.482761 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:19.484745 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:19.485465 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:19.485468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:19.485470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:19.485560 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:19.485941 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:19.485944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:19.485946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:19.486185 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:19.486299 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:19.486937 (+ 638us) server_negotiation.cc:299] Negotiation successful
0929 18:32:19.487104 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
WARNING: no policy specified for kudu/127.28.207.67@KRBTEST.COM; defaulting to no policy
I20260929 18:32:19.488641 426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:19.471252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:37315 (local address 127.28.207.66:55659)
0929 18:32:19.471848 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:19.471913 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:19.473718 (+ 1805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:19.474456 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:19.474466 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:19.474979 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:19.475601 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:19.475617 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:19.476783 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:19.476790 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:19.478169 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:19.478187 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:19.478956 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:19.478971 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:19.479174 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:19.480041 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:19.480059 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:19.482440 (+ 2381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:19.484917 (+ 2477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:19.484926 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:19.484956 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:19.485336 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:19.485665 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:19.485668 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:19.485670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:19.485833 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:19.486296 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:19.486306 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:19.486639 (+ 333us) client_negotiation.cc:769] Sending connection context
0929 18:32:19.486859 (+ 220us) client_negotiation.cc:240] Negotiation successful
0929 18:32:19.487315 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":118,"threads_started":1}
I20260929 18:32:19.490976 424 heartbeater.cc:347] Connected to a master server at 127.28.207.126:37315
I20260929 18:32:19.491379 424 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.67@KRBTEST.COM" created.
I20260929 18:32:19.492246 424 heartbeater.cc:511] Master 127.28.207.126:37315 requested a full tablet report, sending...
I20260929 18:32:19.494074 32567 ts_manager.cc:194] Registered new tserver with Master: 0de0afff64a44deabf1b83e270b42bff (127.28.207.66:33785)
I20260929 18:32:19.494789 32567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.66@KRBTEST.COM'} at 127.28.207.66:55659
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab.
Entry for principal kudu/127.28.207.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab.
I20260929 18:32:19.511353 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.67:0
--local_ip_for_outbound_sockets=127.28.207.67
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:37315
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:43939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:19.624989 429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:19.625267 429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:19.625311 429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:19.630077 429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:19.630206 429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:19.630312 429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.67
I20260929 18:32:19.637424 429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:43939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:37315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.67
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:19.638803 429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:19.640054 429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:19.650998 435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:19.651211 429 server_base.cc:1034] running on GCE node
W20260929 18:32:19.650784 434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:19.650756 437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:19.652222 429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:19.653043 429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:19.654326 429 hybrid_clock.cc:648] HybridClock initialized: now 1790706739654281 us; error 69 us; skew 500 ppm
Sep 29 18:32:19 dist-test-slave-qcpv krb5kdc[32522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706739, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:19.657572 429 init.cc:377] Logged in from keytab as kudu/127.28.207.67@KRBTEST.COM (short username kudu)
I20260929 18:32:19.658612 429 webserver.cc:460] Webserver started at http://127.28.207.67:41427/ using document root <none> and password file <none>
I20260929 18:32:19.659271 429 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:19.659365 429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:19.659600 429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:19.661759 429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-2/data/instance:
uuid: "5434099442474894b9c76b6e9780f0e4"
format_stamp: "Formatted at 2026-09-29 18:32:19 on dist-test-slave-qcpv"
server_key: "1911d5cd1df04cc9e513f7a4621fea71"
server_key_iv: "0c88f8e94723231c70c45c384d5d5d8a"
server_key_version: "encryptionkey@0"
I20260929 18:32:19.662345 429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance:
uuid: "5434099442474894b9c76b6e9780f0e4"
format_stamp: "Formatted at 2026-09-29 18:32:19 on dist-test-slave-qcpv"
server_key: "1911d5cd1df04cc9e513f7a4621fea71"
server_key_iv: "0c88f8e94723231c70c45c384d5d5d8a"
server_key_version: "encryptionkey@0"
I20260929 18:32:19.667140 429 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 18:32:19.670503 444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:19.672483 429 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:32:19.672662 429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-2/wal
uuid: "5434099442474894b9c76b6e9780f0e4"
format_stamp: "Formatted at 2026-09-29 18:32:19 on dist-test-slave-qcpv"
server_key: "1911d5cd1df04cc9e513f7a4621fea71"
server_key_iv: "0c88f8e94723231c70c45c384d5d5d8a"
server_key_version: "encryptionkey@0"
I20260929 18:32:19.672811 429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:19.686268 429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:19.689857 429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:19.690092 429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:19.690755 429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:32:19.691716 429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:19.691793 429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:19.691877 429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:19.691926 429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:19.703079 429 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.67:40611
I20260929 18:32:19.703097 557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.67:40611 every 8 connection(s)
I20260929 18:32:19.704022 429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
I20260929 18:32:19.712399 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 429
I20260929 18:32:19.712533 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance
I20260929 18:32:19.712842 29501 external_mini_cluster.cc:1428] Setting key 333bffe737da66e3cf39dd8e4835c05b
Sep 29 18:32:19 dist-test-slave-qcpv krb5kdc[32522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706739, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:32:19.723272 32606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:19.706644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:60735 (local address 127.28.207.126:37315)
0929 18:32:19.706832 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:19.706836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:19.707839 (+ 1003us) server_negotiation.cc:407] Connection header received
0929 18:32:19.709769 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:19.709832 (+ 63us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:19.709928 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:19.710042 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:19.712089 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:19.712915 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:19.714336 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:19.714562 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:19.714679 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:19.718550 (+ 3871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:19.718556 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:19.718568 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:19.718611 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:19.720601 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:19.721354 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:19.721358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:19.721359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:19.721481 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:19.721876 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:19.721879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:19.721881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:19.722104 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:19.722203 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:19.722831 (+ 628us) server_negotiation.cc:299] Negotiation successful
0929 18:32:19.723106 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 18:32:19.724086 560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:19.706983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:37315 (local address 127.28.207.67:60735)
0929 18:32:19.707595 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:19.707667 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:19.709412 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:19.710332 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:19.710343 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:19.711082 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:19.711877 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:19.711899 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:19.713092 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:19.713095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:19.714204 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:19.714215 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:19.714724 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:19.714727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:19.715008 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:19.716030 (+ 1022us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:19.716048 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:19.718153 (+ 2105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:19.720766 (+ 2613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:19.720772 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:19.720789 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:19.721180 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:19.721577 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:19.721579 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:19.721581 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:19.721724 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:19.722223 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:19.722229 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:19.722501 (+ 272us) client_negotiation.cc:769] Sending connection context
0929 18:32:19.722677 (+ 176us) client_negotiation.cc:240] Negotiation successful
0929 18:32:19.723076 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":136,"threads_started":1}
I20260929 18:32:19.725839 558 heartbeater.cc:347] Connected to a master server at 127.28.207.126:37315
I20260929 18:32:19.726171 558 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:19.726743 558 heartbeater.cc:511] Master 127.28.207.126:37315 requested a full tablet report, sending...
I20260929 18:32:19.728341 32567 ts_manager.cc:194] Registered new tserver with Master: 5434099442474894b9c76b6e9780f0e4 (127.28.207.67:40611)
I20260929 18:32:19.728943 29501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:32:19.729249 32567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.67@KRBTEST.COM'} at 127.28.207.67:60735
Sep 29 18:32:19 dist-test-slave-qcpv krb5kdc[32522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706739, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 18:32:19 dist-test-slave-qcpv krb5kdc[32522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706739, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:32:19.762619 32606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:19.749899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42722 (local address 127.28.207.126:37315)
0929 18:32:19.750124 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:19.750127 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:19.750542 (+ 415us) server_negotiation.cc:407] Connection header received
0929 18:32:19.750899 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:19.750906 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:19.750972 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:19.751081 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:19.752365 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:19.753255 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:19.754732 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:19.755102 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:19.755231 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:19.758604 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:19.758608 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:19.758610 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:19.758640 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:19.760316 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:19.760963 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:19.760966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:19.760967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:19.761052 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:19.761451 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:19.761453 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:19.761454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:19.761709 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:19.761849 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:19.762259 (+ 410us) server_negotiation.cc:299] Negotiation successful
0929 18:32:19.762380 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260929 18:32:19.763226 32567 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42722
I20260929 18:32:19.764516 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 32629
I20260929 18:32:19.771847 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 32763
I20260929 18:32:19.779512 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 429
I20260929 18:32:19.786717 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 32535
2026-09-29T18:32:19Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2405 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:32:19 dist-test-slave-qcpv krb5kdc[570](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:32:19 dist-test-slave-qcpv krb5kdc[570](info): set up 2 sockets
Sep 29 18:32:19 dist-test-slave-qcpv krb5kdc[570](info): commencing operation
krb5kdc: starting...
W20260929 18:32:20.863639 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:32:20 dist-test-slave-qcpv krb5kdc[570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:32:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:32:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.207.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
I20260929 18:32:21.006692 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:34141
--webserver_interface=127.28.207.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:35555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:34141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:21.169745 583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:21.170194 583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:21.170295 583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:21.175205 583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:32:21.175370 583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:21.175410 583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:21.175443 583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:32:21.175482 583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:32:21.182240 583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:35555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:34141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:34141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:21.184041 583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:21.185415 583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:21.196017 591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:21.196007 589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:21.196007 588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:21.196199 583 server_base.cc:1034] running on GCE node
I20260929 18:32:21.197333 583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:21.198971 583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:21.200198 583 hybrid_clock.cc:648] HybridClock initialized: now 1790706741200151 us; error 64 us; skew 500 ppm
Sep 29 18:32:21 dist-test-slave-qcpv krb5kdc[570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706741, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:21.203881 583 init.cc:377] Logged in from keytab as kudu/127.28.207.126@KRBTEST.COM (short username kudu)
I20260929 18:32:21.205125 583 webserver.cc:460] Webserver started at http://127.28.207.126:44321/ using document root <none> and password file <none>
I20260929 18:32:21.206008 583 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:21.206115 583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:21.206395 583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:21.209578 583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "aab2d87ead40464b8fb2593584aeb550"
format_stamp: "Formatted at 2026-09-29 18:32:21 on dist-test-slave-qcpv"
server_key: "a2c3c48a5cbf9b0b61a2a6c9b574f1bc"
server_key_iv: "8b70fcaacc6823d372c641b162d4863c"
server_key_version: "encryptionkey@0"
I20260929 18:32:21.210285 583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "aab2d87ead40464b8fb2593584aeb550"
format_stamp: "Formatted at 2026-09-29 18:32:21 on dist-test-slave-qcpv"
server_key: "a2c3c48a5cbf9b0b61a2a6c9b574f1bc"
server_key_iv: "8b70fcaacc6823d372c641b162d4863c"
server_key_version: "encryptionkey@0"
I20260929 18:32:21.215138 583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 18:32:21.218428 598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:21.220252 583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:32:21.220481 583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "aab2d87ead40464b8fb2593584aeb550"
format_stamp: "Formatted at 2026-09-29 18:32:21 on dist-test-slave-qcpv"
server_key: "a2c3c48a5cbf9b0b61a2a6c9b574f1bc"
server_key_iv: "8b70fcaacc6823d372c641b162d4863c"
server_key_version: "encryptionkey@0"
I20260929 18:32:21.220647 583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:21.275689 583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:21.280526 583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:21.280819 583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:21.291662 583 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:34141
I20260929 18:32:21.291697 650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:34141 every 8 connection(s)
I20260929 18:32:21.293159 583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:32:21.296806 651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:21.298533 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 583
I20260929 18:32:21.298683 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:32:21.299014 29501 external_mini_cluster.cc:1428] Setting key 88e9eea07695b1214b888ce39f5edb96
I20260929 18:32:21.303735 651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aab2d87ead40464b8fb2593584aeb550: Bootstrap starting.
I20260929 18:32:21.306931 651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aab2d87ead40464b8fb2593584aeb550: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:21.307989 651 log.cc:824] T 00000000000000000000000000000000 P aab2d87ead40464b8fb2593584aeb550: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:32:21 dist-test-slave-qcpv krb5kdc[570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:32:21.310869 651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aab2d87ead40464b8fb2593584aeb550: No bootstrap required, opened a new log
I20260929 18:32:21.316608 651 raft_consensus.cc:348] T 00000000000000000000000000000000 P aab2d87ead40464b8fb2593584aeb550 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab2d87ead40464b8fb2593584aeb550" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 34141 } }
I20260929 18:32:21.316989 651 raft_consensus.cc:374] T 00000000000000000000000000000000 P aab2d87ead40464b8fb2593584aeb550 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:21.317050 651 raft_consensus.cc:729] T 00000000000000000000000000000000 P aab2d87ead40464b8fb2593584aeb550 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aab2d87ead40464b8fb2593584aeb550, State: Initialized, Role: FOLLOWER
I20260929 18:32:21.317771 651 consensus_queue.cc:260] T 00000000000000000000000000000000 P aab2d87ead40464b8fb2593584aeb550 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab2d87ead40464b8fb2593584aeb550" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 34141 } }
I20260929 18:32:21.317966 651 raft_consensus.cc:388] T 00000000000000000000000000000000 P aab2d87ead40464b8fb2593584aeb550 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:32:21.318035 651 raft_consensus.cc:482] T 00000000000000000000000000000000 P aab2d87ead40464b8fb2593584aeb550 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:32:21.318131 651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aab2d87ead40464b8fb2593584aeb550 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:21.318298 654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:21.300966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43734 (local address 127.28.207.126:34141)
0929 18:32:21.301574 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:21.301597 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:21.301659 (+ 62us) server_negotiation.cc:407] Connection header received
0929 18:32:21.302954 (+ 1295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:21.302981 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:21.303607 (+ 626us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:21.304084 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:21.305111 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:21.306307 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:21.307795 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:21.308507 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:21.308742 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:21.311493 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:21.311501 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:21.311523 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:21.311563 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:21.314025 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:21.314654 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:21.314660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:21.314668 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:21.314843 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:21.315236 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:21.315239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:21.315241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:21.315926 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:21.316162 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:21.316498 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 18:32:21.316892 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":160,"threads_started":1}
I20260929 18:32:21.319703 651 raft_consensus.cc:504] T 00000000000000000000000000000000 P aab2d87ead40464b8fb2593584aeb550 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab2d87ead40464b8fb2593584aeb550" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 34141 } }
I20260929 18:32:21.320137 651 leader_election.cc:302] T 00000000000000000000000000000000 P aab2d87ead40464b8fb2593584aeb550 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aab2d87ead40464b8fb2593584aeb550; no voters:
I20260929 18:32:21.320580 651 leader_election.cc:288] T 00000000000000000000000000000000 P aab2d87ead40464b8fb2593584aeb550 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:32:21.320755 656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aab2d87ead40464b8fb2593584aeb550 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:32:21.321055 656 raft_consensus.cc:686] T 00000000000000000000000000000000 P aab2d87ead40464b8fb2593584aeb550 [term 1 LEADER]: Becoming Leader. State: Replica: aab2d87ead40464b8fb2593584aeb550, State: Running, Role: LEADER
I20260929 18:32:21.321471 656 consensus_queue.cc:237] T 00000000000000000000000000000000 P aab2d87ead40464b8fb2593584aeb550 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab2d87ead40464b8fb2593584aeb550" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 34141 } }
I20260929 18:32:21.322445 651 sys_catalog.cc:564] T 00000000000000000000000000000000 P aab2d87ead40464b8fb2593584aeb550 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:32:21.323657 657 sys_catalog.cc:455] T 00000000000000000000000000000000 P aab2d87ead40464b8fb2593584aeb550 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aab2d87ead40464b8fb2593584aeb550" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab2d87ead40464b8fb2593584aeb550" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 34141 } } }
I20260929 18:32:21.323846 657 sys_catalog.cc:458] T 00000000000000000000000000000000 P aab2d87ead40464b8fb2593584aeb550 [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:21.324317 665 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:32:21.325038 658 sys_catalog.cc:455] T 00000000000000000000000000000000 P aab2d87ead40464b8fb2593584aeb550 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aab2d87ead40464b8fb2593584aeb550. Latest consensus state: current_term: 1 leader_uuid: "aab2d87ead40464b8fb2593584aeb550" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aab2d87ead40464b8fb2593584aeb550" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 34141 } } }
I20260929 18:32:21.325163 658 sys_catalog.cc:458] T 00000000000000000000000000000000 P aab2d87ead40464b8fb2593584aeb550 [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:21.327726 665 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:32:21.335094 665 catalog_manager.cc:1347] Generated new cluster ID: f4da12ecc7cc4792916f2db3e2ef5ea7
I20260929 18:32:21.335199 665 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:32:21.371114 665 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:32:21.372469 665 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:32:21.377470 665 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aab2d87ead40464b8fb2593584aeb550: Generated new TSK 0
I20260929 18:32:21.378365 665 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.207.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab.
Entry for principal kudu/127.28.207.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab.
I20260929 18:32:21.419147 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.65:0
--local_ip_for_outbound_sockets=127.28.207.65
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:34141
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:35555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:21.571959 677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:21.572275 677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:21.572324 677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:21.576612 677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:21.576709 677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:21.576781 677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.65
I20260929 18:32:21.582576 677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:35555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:34141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.65
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:21.584110 677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:21.585618 677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:21.597200 683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:21.597265 685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:21.597470 677 server_base.cc:1034] running on GCE node
W20260929 18:32:21.597386 682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:21.598222 677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:21.599021 677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:21.600242 677 hybrid_clock.cc:648] HybridClock initialized: now 1790706741600215 us; error 48 us; skew 500 ppm
Sep 29 18:32:21 dist-test-slave-qcpv krb5kdc[570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706741, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:21.603816 677 init.cc:377] Logged in from keytab as kudu/127.28.207.65@KRBTEST.COM (short username kudu)
I20260929 18:32:21.604900 677 webserver.cc:460] Webserver started at http://127.28.207.65:43965/ using document root <none> and password file <none>
I20260929 18:32:21.605544 677 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:21.605644 677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:21.605989 677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:21.608500 677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-0/data/instance:
uuid: "8f12b60f649e4493950d5149f127bae3"
format_stamp: "Formatted at 2026-09-29 18:32:21 on dist-test-slave-qcpv"
server_key: "99bb2bd4ee8ad4c143679107bce7180a"
server_key_iv: "e0da7a20451154b3be51312f9e7f33b6"
server_key_version: "encryptionkey@0"
I20260929 18:32:21.609154 677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance:
uuid: "8f12b60f649e4493950d5149f127bae3"
format_stamp: "Formatted at 2026-09-29 18:32:21 on dist-test-slave-qcpv"
server_key: "99bb2bd4ee8ad4c143679107bce7180a"
server_key_iv: "e0da7a20451154b3be51312f9e7f33b6"
server_key_version: "encryptionkey@0"
I20260929 18:32:21.613976 677 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:32:21.617137 692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:21.619436 677 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 18:32:21.619611 677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-0/wal
uuid: "8f12b60f649e4493950d5149f127bae3"
format_stamp: "Formatted at 2026-09-29 18:32:21 on dist-test-slave-qcpv"
server_key: "99bb2bd4ee8ad4c143679107bce7180a"
server_key_iv: "e0da7a20451154b3be51312f9e7f33b6"
server_key_version: "encryptionkey@0"
I20260929 18:32:21.619748 677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:21.637893 677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:21.641975 677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:21.642243 677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:21.643016 677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:32:21.643975 677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:21.644057 677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:21.644155 677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:21.644210 677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:21.656903 677 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.65:38609
I20260929 18:32:21.657097 805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.65:38609 every 8 connection(s)
I20260929 18:32:21.658059 677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
I20260929 18:32:21.659533 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 677
I20260929 18:32:21.659677 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance
I20260929 18:32:21.660036 29501 external_mini_cluster.cc:1428] Setting key b39101fec4a0feeb694dbb2d96cd3220
WARNING: no policy specified for kudu/127.28.207.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.66@KRBTEST.COM" created.
Sep 29 18:32:21 dist-test-slave-qcpv krb5kdc[570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706741, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:32:21.685113 654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:21.660971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:54949 (local address 127.28.207.126:34141)
0929 18:32:21.661172 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:21.661176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:21.662440 (+ 1264us) server_negotiation.cc:407] Connection header received
0929 18:32:21.664271 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:21.664276 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:21.664364 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:21.664492 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:21.667031 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:21.667806 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:21.670631 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:21.671139 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:21.671273 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:21.679120 (+ 7847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:21.679124 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:21.679126 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:21.679159 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:21.681417 (+ 2258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:21.682365 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:21.682372 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:21.682376 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:21.682503 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:21.683248 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:21.683254 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:21.683258 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:21.683581 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:21.683712 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:21.684550 (+ 838us) server_negotiation.cc:299] Negotiation successful
0929 18:32:21.684776 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 18:32:21.686291 808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:21.661524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:34141 (local address 127.28.207.65:54949)
0929 18:32:21.662216 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:21.662283 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:21.663963 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:21.664682 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:21.664693 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:21.665726 (+ 1033us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:21.666679 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:21.666724 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:21.668134 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:21.668138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:21.670379 (+ 2241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:21.670400 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:21.671402 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:21.671408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:21.671709 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:21.672655 (+ 946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:21.672688 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:21.678787 (+ 6099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:21.681675 (+ 2888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:21.681684 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:21.681708 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:21.682195 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:21.682663 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:21.682666 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:21.682667 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:21.683008 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:21.683743 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:21.683751 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:21.684156 (+ 405us) client_negotiation.cc:769] Sending connection context
0929 18:32:21.684437 (+ 281us) client_negotiation.cc:240] Negotiation successful
0929 18:32:21.685027 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":216,"threads_started":1}
I20260929 18:32:21.688110 806 heartbeater.cc:347] Connected to a master server at 127.28.207.126:34141
I20260929 18:32:21.688503 806 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:21.689298 806 heartbeater.cc:511] Master 127.28.207.126:34141 requested a full tablet report, sending...
I20260929 18:32:21.691756 615 ts_manager.cc:194] Registered new tserver with Master: 8f12b60f649e4493950d5149f127bae3 (127.28.207.65:38609)
I20260929 18:32:21.694212 615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.65@KRBTEST.COM'} at 127.28.207.65:54949
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab.
Entry for principal kudu/127.28.207.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab.
I20260929 18:32:21.700161 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.66:0
--local_ip_for_outbound_sockets=127.28.207.66
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:34141
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:35555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:21.824694 811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:21.824985 811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:21.825078 811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:21.829584 811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:21.829669 811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:21.829734 811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.66
I20260929 18:32:21.835290 811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:35555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:34141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.66
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:21.836473 811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:21.837633 811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:21.848697 819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:21.849001 816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:21.848973 811 server_base.cc:1034] running on GCE node
W20260929 18:32:21.849228 817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:21.850041 811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:21.851176 811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:21.852428 811 hybrid_clock.cc:648] HybridClock initialized: now 1790706741852385 us; error 69 us; skew 500 ppm
Sep 29 18:32:21 dist-test-slave-qcpv krb5kdc[570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706741, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:21.856691 811 init.cc:377] Logged in from keytab as kudu/127.28.207.66@KRBTEST.COM (short username kudu)
I20260929 18:32:21.858039 811 webserver.cc:460] Webserver started at http://127.28.207.66:38017/ using document root <none> and password file <none>
I20260929 18:32:21.859050 811 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:21.859180 811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:21.859472 811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:21.862547 811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-1/data/instance:
uuid: "a164495d183643fe92bed2c9f45896f0"
format_stamp: "Formatted at 2026-09-29 18:32:21 on dist-test-slave-qcpv"
server_key: "b4544809c5704c9e489ac65f076d705d"
server_key_iv: "5de7261ad4c0fb6b752ed3b7ee159df0"
server_key_version: "encryptionkey@0"
I20260929 18:32:21.863549 811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance:
uuid: "a164495d183643fe92bed2c9f45896f0"
format_stamp: "Formatted at 2026-09-29 18:32:21 on dist-test-slave-qcpv"
server_key: "b4544809c5704c9e489ac65f076d705d"
server_key_iv: "5de7261ad4c0fb6b752ed3b7ee159df0"
server_key_version: "encryptionkey@0"
I20260929 18:32:21.868775 811 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 18:32:21.872725 826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:21.874545 811 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 18:32:21.874758 811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-1/wal
uuid: "a164495d183643fe92bed2c9f45896f0"
format_stamp: "Formatted at 2026-09-29 18:32:21 on dist-test-slave-qcpv"
server_key: "b4544809c5704c9e489ac65f076d705d"
server_key_iv: "5de7261ad4c0fb6b752ed3b7ee159df0"
server_key_version: "encryptionkey@0"
I20260929 18:32:21.874984 811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:21.906594 811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:21.911360 811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:21.911669 811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:21.912616 811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:32:21.913746 811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:21.913828 811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:21.913909 811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:21.913955 811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:21.928506 811 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.66:35659
I20260929 18:32:21.928710 939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.66:35659 every 8 connection(s)
I20260929 18:32:21.930056 811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
I20260929 18:32:21.940384 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 811
I20260929 18:32:21.940539 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance
I20260929 18:32:21.941062 29501 external_mini_cluster.cc:1428] Setting key 9e7e6223ef5a66b462b0ec752d475a77
Sep 29 18:32:21 dist-test-slave-qcpv krb5kdc[570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706741, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:32:21.947497 654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:21.932449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:47207 (local address 127.28.207.126:34141)
0929 18:32:21.932703 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:21.932709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:21.933974 (+ 1265us) server_negotiation.cc:407] Connection header received
0929 18:32:21.935531 (+ 1557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:21.935535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:21.935609 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:21.935719 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:21.937186 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:21.938107 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:21.939617 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:21.939888 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:21.940083 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:21.943707 (+ 3624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:21.943710 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:21.943713 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:21.943736 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:21.945266 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:21.945856 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:21.945866 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:21.945868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:21.945937 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:21.946289 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:21.946292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:21.946294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:21.946491 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:21.946601 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:21.947190 (+ 589us) server_negotiation.cc:299] Negotiation successful
0929 18:32:21.947318 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 18:32:21.948360 942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:21.933086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:34141 (local address 127.28.207.66:47207)
0929 18:32:21.933765 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:21.933824 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:21.935254 (+ 1430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:21.935949 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:21.935957 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:21.936387 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:21.936927 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:21.936940 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:21.938271 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:21.938274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:21.939441 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:21.939452 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:21.940150 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:21.940153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:21.940312 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:21.941137 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:21.941155 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:21.943458 (+ 2303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:21.945411 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:21.945418 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:21.945436 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:21.945737 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:21.946029 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:21.946034 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:21.946036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:21.946183 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:21.946598 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:21.946606 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:21.946935 (+ 329us) client_negotiation.cc:769] Sending connection context
0929 18:32:21.947125 (+ 190us) client_negotiation.cc:240] Negotiation successful
0929 18:32:21.947465 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":145,"threads_started":1}
I20260929 18:32:21.949719 940 heartbeater.cc:347] Connected to a master server at 127.28.207.126:34141
I20260929 18:32:21.949981 940 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:21.950492 940 heartbeater.cc:511] Master 127.28.207.126:34141 requested a full tablet report, sending...
I20260929 18:32:21.951866 615 ts_manager.cc:194] Registered new tserver with Master: a164495d183643fe92bed2c9f45896f0 (127.28.207.66:35659)
I20260929 18:32:21.952888 615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.66@KRBTEST.COM'} at 127.28.207.66:47207
WARNING: no policy specified for kudu/127.28.207.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab.
Entry for principal kudu/127.28.207.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab.
I20260929 18:32:21.978096 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.67:0
--local_ip_for_outbound_sockets=127.28.207.67
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:34141
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:35555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:22.125352 945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:22.125725 945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:22.125810 945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:22.130657 945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:22.130759 945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:22.130832 945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.67
I20260929 18:32:22.140589 945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:35555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:34141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.67
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:22.142087 945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:22.143467 945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:22.156399 953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:22.156472 950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:22.156402 951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:22.157162 945 server_base.cc:1034] running on GCE node
I20260929 18:32:22.157823 945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:22.158754 945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:22.162149 945 hybrid_clock.cc:648] HybridClock initialized: now 1790706742162112 us; error 81 us; skew 500 ppm
Sep 29 18:32:22 dist-test-slave-qcpv krb5kdc[570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706742, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:22.165967 945 init.cc:377] Logged in from keytab as kudu/127.28.207.67@KRBTEST.COM (short username kudu)
I20260929 18:32:22.167510 945 webserver.cc:460] Webserver started at http://127.28.207.67:38151/ using document root <none> and password file <none>
I20260929 18:32:22.168272 945 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:22.168386 945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:22.168665 945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:22.171830 945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-2/data/instance:
uuid: "aa35852ecd29460590d355f78869f266"
format_stamp: "Formatted at 2026-09-29 18:32:22 on dist-test-slave-qcpv"
server_key: "b49a9a081d4af19057a643d1174cce0c"
server_key_iv: "185a3c318d790bfde667e565dc2e42c1"
server_key_version: "encryptionkey@0"
I20260929 18:32:22.172590 945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance:
uuid: "aa35852ecd29460590d355f78869f266"
format_stamp: "Formatted at 2026-09-29 18:32:22 on dist-test-slave-qcpv"
server_key: "b49a9a081d4af19057a643d1174cce0c"
server_key_iv: "185a3c318d790bfde667e565dc2e42c1"
server_key_version: "encryptionkey@0"
I20260929 18:32:22.178220 945 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 18:32:22.181944 960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:22.183966 945 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:32:22.184171 945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-2/wal
uuid: "aa35852ecd29460590d355f78869f266"
format_stamp: "Formatted at 2026-09-29 18:32:22 on dist-test-slave-qcpv"
server_key: "b49a9a081d4af19057a643d1174cce0c"
server_key_iv: "185a3c318d790bfde667e565dc2e42c1"
server_key_version: "encryptionkey@0"
I20260929 18:32:22.184347 945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:22.220422 945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:22.226046 945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:22.226759 945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:22.227671 945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:32:22.229228 945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:22.229317 945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:22.229405 945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:22.229458 945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:22.249549 945 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.67:41687
I20260929 18:32:22.249651 1073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.67:41687 every 8 connection(s)
I20260929 18:32:22.252408 945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
I20260929 18:32:22.259191 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 945
I20260929 18:32:22.259361 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance
I20260929 18:32:22.259709 29501 external_mini_cluster.cc:1428] Setting key 9eb0b0223760dbba7d8c69fb3d66e426
Sep 29 18:32:22 dist-test-slave-qcpv krb5kdc[570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706742, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:32:22.274533 654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:22.255670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:52995 (local address 127.28.207.126:34141)
0929 18:32:22.255838 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:22.255843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:22.257689 (+ 1846us) server_negotiation.cc:407] Connection header received
0929 18:32:22.258932 (+ 1243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:22.258938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:22.259025 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:22.259193 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:22.261485 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:22.262242 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:22.263925 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:22.264164 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:22.264286 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:22.268346 (+ 4060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:22.268350 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:22.268426 (+ 76us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:22.268470 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:22.270739 (+ 2269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:22.271713 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:22.271717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:22.271719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:22.271849 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:22.272710 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:22.272714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:22.272715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:22.273092 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:22.273222 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:22.274249 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0929 18:32:22.274397 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 18:32:22.276147 1076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:22.256242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:34141 (local address 127.28.207.67:52995)
0929 18:32:22.257423 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:22.257498 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:22.258595 (+ 1097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:22.259570 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:22.259580 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:22.260161 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:22.261255 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:22.261279 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:22.262518 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:22.262523 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:22.263760 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:22.263773 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:22.264403 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:22.264407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:22.264603 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:22.265444 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:22.265464 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:22.268019 (+ 2555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:22.271052 (+ 3033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:22.271061 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:22.271085 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:22.271553 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:22.272029 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:22.272032 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:22.272034 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:22.272372 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:22.273258 (+ 886us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:22.273266 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:22.273792 (+ 526us) client_negotiation.cc:769] Sending connection context
0929 18:32:22.274089 (+ 297us) client_negotiation.cc:240] Negotiation successful
0929 18:32:22.274662 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":863,"thread_start_us":599,"threads_started":1}
I20260929 18:32:22.278295 1074 heartbeater.cc:347] Connected to a master server at 127.28.207.126:34141
I20260929 18:32:22.278687 1074 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:22.279402 1074 heartbeater.cc:511] Master 127.28.207.126:34141 requested a full tablet report, sending...
I20260929 18:32:22.280651 615 ts_manager.cc:194] Registered new tserver with Master: aa35852ecd29460590d355f78869f266 (127.28.207.67:41687)
I20260929 18:32:22.281481 615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.67@KRBTEST.COM'} at 127.28.207.67:52995
I20260929 18:32:22.289108 29501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 18:32:22 dist-test-slave-qcpv krb5kdc[570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 18:32:22 dist-test-slave-qcpv krb5kdc[570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.207.65@KRBTEST.COM
I20260929 18:32:22.324813 1081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:22.310739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35268 (local address 127.28.207.65:38609)
0929 18:32:22.311262 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:22.311270 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:22.311308 (+ 38us) server_negotiation.cc:407] Connection header received
0929 18:32:22.311379 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:22.311385 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:22.311664 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:22.311851 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:22.313012 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:22.314164 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:22.315343 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:22.316003 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:22.316145 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:22.319613 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:22.319633 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:22.319649 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:22.319689 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:22.321796 (+ 2107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:22.322668 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:22.322674 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:22.322676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:22.322770 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:22.323126 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:22.323130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:22.323131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:22.323650 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:22.324048 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:22.324299 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 18:32:22.324510 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":176,"threads_started":1}
Sep 29 18:32:22 dist-test-slave-qcpv krb5kdc[570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:32:22.340412 654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:22.328677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43748 (local address 127.28.207.126:34141)
0929 18:32:22.328958 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:22.328964 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:22.329125 (+ 161us) server_negotiation.cc:407] Connection header received
0929 18:32:22.329354 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:22.329358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:22.329442 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:22.329552 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:22.330402 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:22.331107 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:22.332054 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:22.332344 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:22.332471 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:22.336310 (+ 3839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:22.336315 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:22.336317 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:22.336357 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:22.338021 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:22.338534 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:22.338538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:22.338540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:22.338687 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:22.339036 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:22.339039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:22.339041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:22.339317 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:22.339611 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:22.339885 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 18:32:22.340169 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
W20260929 18:32:22.341015 615 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43748
I20260929 18:32:22.341961 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 677
I20260929 18:32:22.348282 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 811
I20260929 18:32:22.354848 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 945
I20260929 18:32:22.360925 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 583
2026-09-29T18:32:22Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2572 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:32:22 dist-test-slave-qcpv krb5kdc[1086](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:32:22 dist-test-slave-qcpv krb5kdc[1086](info): set up 2 sockets
Sep 29 18:32:22 dist-test-slave-qcpv krb5kdc[1086](info): commencing operation
krb5kdc: starting...
W20260929 18:32:23.435384 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:32:23 dist-test-slave-qcpv krb5kdc[1086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:32:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:32:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.207.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
I20260929 18:32:23.575049 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:40703
--webserver_interface=127.28.207.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:36699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:40703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:23.724001 1099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:23.724499 1099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:23.724952 1099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:23.730541 1099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:32:23.730698 1099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:23.730736 1099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:23.730780 1099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:32:23.730816 1099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:32:23.737744 1099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:36699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:40703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:40703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:23.739398 1099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:23.740757 1099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:23.751380 1107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:23.751845 1105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:23.752067 1104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:23.752696 1099 server_base.cc:1034] running on GCE node
I20260929 18:32:23.753567 1099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:23.755609 1099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:23.756991 1099 hybrid_clock.cc:648] HybridClock initialized: now 1790706743756929 us; error 81 us; skew 500 ppm
Sep 29 18:32:23 dist-test-slave-qcpv krb5kdc[1086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706743, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:23.760298 1099 init.cc:377] Logged in from keytab as kudu/127.28.207.126@KRBTEST.COM (short username kudu)
I20260929 18:32:23.761181 1099 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:23.761256 1099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:23.761458 1099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:23.763955 1099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "1fed4d7ba6804091a3941424286bc421"
format_stamp: "Formatted at 2026-09-29 18:32:23 on dist-test-slave-qcpv"
server_key: "4dd64337db3542877a7c0adca8d5394c"
server_key_iv: "79f5f34b78bfdd71b00e916fe53e8f52"
server_key_version: "encryptionkey@0"
I20260929 18:32:23.764516 1099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "1fed4d7ba6804091a3941424286bc421"
format_stamp: "Formatted at 2026-09-29 18:32:23 on dist-test-slave-qcpv"
server_key: "4dd64337db3542877a7c0adca8d5394c"
server_key_iv: "79f5f34b78bfdd71b00e916fe53e8f52"
server_key_version: "encryptionkey@0"
I20260929 18:32:23.769387 1099 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 18:32:23.774763 1113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:23.777035 1099 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260929 18:32:23.777331 1099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "1fed4d7ba6804091a3941424286bc421"
format_stamp: "Formatted at 2026-09-29 18:32:23 on dist-test-slave-qcpv"
server_key: "4dd64337db3542877a7c0adca8d5394c"
server_key_iv: "79f5f34b78bfdd71b00e916fe53e8f52"
server_key_version: "encryptionkey@0"
I20260929 18:32:23.777521 1099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:23.800719 1099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:23.805341 1099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:23.805626 1099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:23.816092 1099 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:40703
I20260929 18:32:23.816444 1165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:40703 every 8 connection(s)
I20260929 18:32:23.817122 1099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:32:23.820541 1166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:23.823642 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 1099
I20260929 18:32:23.823879 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:32:23.824200 29501 external_mini_cluster.cc:1428] Setting key 67fc691df11f68ad505620f682ff1366
I20260929 18:32:23.828673 1166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fed4d7ba6804091a3941424286bc421: Bootstrap starting.
I20260929 18:32:23.831439 1166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fed4d7ba6804091a3941424286bc421: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:23.832422 1166 log.cc:824] T 00000000000000000000000000000000 P 1fed4d7ba6804091a3941424286bc421: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:23.835597 1166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fed4d7ba6804091a3941424286bc421: No bootstrap required, opened a new log
Sep 29 18:32:23 dist-test-slave-qcpv krb5kdc[1086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:32:23.839775 1166 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fed4d7ba6804091a3941424286bc421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fed4d7ba6804091a3941424286bc421" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 40703 } }
I20260929 18:32:23.840065 1166 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fed4d7ba6804091a3941424286bc421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:23.840118 1166 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fed4d7ba6804091a3941424286bc421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fed4d7ba6804091a3941424286bc421, State: Initialized, Role: FOLLOWER
I20260929 18:32:23.840720 1166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fed4d7ba6804091a3941424286bc421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fed4d7ba6804091a3941424286bc421" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 40703 } }
I20260929 18:32:23.840898 1166 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fed4d7ba6804091a3941424286bc421 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:32:23.841001 1166 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fed4d7ba6804091a3941424286bc421 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:32:23.841133 1166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fed4d7ba6804091a3941424286bc421 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:23.844928 1169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:23.826339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38750 (local address 127.28.207.126:40703)
0929 18:32:23.826977 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:23.827004 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:23.827053 (+ 49us) server_negotiation.cc:407] Connection header received
0929 18:32:23.827948 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:23.827973 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:23.828477 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:23.828971 (+ 494us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:23.829899 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:23.831653 (+ 1754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:23.832982 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:23.833771 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:23.833944 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:23.837855 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:23.837862 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:23.837884 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:23.837925 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:23.840662 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:23.841376 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:23.841382 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:23.841388 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:23.841535 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:23.841852 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:23.841856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:23.841858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:23.842404 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:23.842658 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:23.843009 (+ 351us) server_negotiation.cc:299] Negotiation successful
0929 18:32:23.843387 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":191,"threads_started":1}
I20260929 18:32:23.893842 1166 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fed4d7ba6804091a3941424286bc421 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fed4d7ba6804091a3941424286bc421" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 40703 } }
I20260929 18:32:23.894457 1166 leader_election.cc:302] T 00000000000000000000000000000000 P 1fed4d7ba6804091a3941424286bc421 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fed4d7ba6804091a3941424286bc421; no voters:
I20260929 18:32:23.894992 1166 leader_election.cc:288] T 00000000000000000000000000000000 P 1fed4d7ba6804091a3941424286bc421 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:32:23.895493 1172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fed4d7ba6804091a3941424286bc421 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:32:23.896027 1172 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fed4d7ba6804091a3941424286bc421 [term 1 LEADER]: Becoming Leader. State: Replica: 1fed4d7ba6804091a3941424286bc421, State: Running, Role: LEADER
I20260929 18:32:23.896704 1166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fed4d7ba6804091a3941424286bc421 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:32:23.896718 1172 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fed4d7ba6804091a3941424286bc421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fed4d7ba6804091a3941424286bc421" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 40703 } }
I20260929 18:32:23.899398 1171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fed4d7ba6804091a3941424286bc421 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fed4d7ba6804091a3941424286bc421. Latest consensus state: current_term: 1 leader_uuid: "1fed4d7ba6804091a3941424286bc421" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fed4d7ba6804091a3941424286bc421" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 40703 } } }
I20260929 18:32:23.900043 1171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fed4d7ba6804091a3941424286bc421 [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:23.899348 1173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fed4d7ba6804091a3941424286bc421 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fed4d7ba6804091a3941424286bc421" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fed4d7ba6804091a3941424286bc421" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 40703 } } }
I20260929 18:32:23.900302 1173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fed4d7ba6804091a3941424286bc421 [sys.catalog]: This master's current role is: LEADER
W20260929 18:32:23.903852 1185 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1fed4d7ba6804091a3941424286bc421: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:32:23.903995 1185 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:32:23.904242 1186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:32:23.905233 1186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:32:23.912905 1186 catalog_manager.cc:1347] Generated new cluster ID: da951da2c1de40f2b4f9109a53a6fe47
I20260929 18:32:23.913044 1186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:32:23.931200 1186 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:32:23.932785 1186 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:32:23.942003 1186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fed4d7ba6804091a3941424286bc421: Generated new TSK 0
I20260929 18:32:23.943040 1186 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.207.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab.
Entry for principal kudu/127.28.207.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab.
W20260929 18:32:25.089362 29501 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.28.207.65: real 1.136s user 0.000s sys 0.005s
I20260929 18:32:25.090204 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.65:0
--local_ip_for_outbound_sockets=127.28.207.65
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:40703
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:36699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:25.223241 1192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:25.223619 1192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:25.223671 1192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:25.228161 1192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:25.228247 1192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:25.228313 1192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.65
I20260929 18:32:25.234156 1192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:36699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:40703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.65
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:25.235481 1192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:25.236577 1192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:25.247707 1198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:25.247773 1200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:25.247707 1197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:25.248328 1192 server_base.cc:1034] running on GCE node
I20260929 18:32:25.248940 1192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:25.249848 1192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:25.251053 1192 hybrid_clock.cc:648] HybridClock initialized: now 1790706745251017 us; error 56 us; skew 500 ppm
Sep 29 18:32:25 dist-test-slave-qcpv krb5kdc[1086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706745, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:25.254827 1192 init.cc:377] Logged in from keytab as kudu/127.28.207.65@KRBTEST.COM (short username kudu)
I20260929 18:32:25.256074 1192 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:25.256206 1192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:25.256508 1192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:25.259629 1192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-0/data/instance:
uuid: "7656fb48bc0f4b68bf088a82abff9728"
format_stamp: "Formatted at 2026-09-29 18:32:25 on dist-test-slave-qcpv"
server_key: "52fd2be591a80b7c82fe4bd0fdc145c6"
server_key_iv: "4c2e775fd2aae742ac5fdb65adbdb033"
server_key_version: "encryptionkey@0"
I20260929 18:32:25.260440 1192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance:
uuid: "7656fb48bc0f4b68bf088a82abff9728"
format_stamp: "Formatted at 2026-09-29 18:32:25 on dist-test-slave-qcpv"
server_key: "52fd2be591a80b7c82fe4bd0fdc145c6"
server_key_iv: "4c2e775fd2aae742ac5fdb65adbdb033"
server_key_version: "encryptionkey@0"
I20260929 18:32:25.266492 1192 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 18:32:25.270426 1206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:25.272713 1192 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:32:25.272930 1192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-0/wal
uuid: "7656fb48bc0f4b68bf088a82abff9728"
format_stamp: "Formatted at 2026-09-29 18:32:25 on dist-test-slave-qcpv"
server_key: "52fd2be591a80b7c82fe4bd0fdc145c6"
server_key_iv: "4c2e775fd2aae742ac5fdb65adbdb033"
server_key_version: "encryptionkey@0"
I20260929 18:32:25.273133 1192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:25.298652 1192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:25.302922 1192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:25.303182 1192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:25.303972 1192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:32:25.305030 1192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:25.305109 1192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:25.305190 1192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:25.305235 1192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:25.318573 1192 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.65:34791
I20260929 18:32:25.318598 1319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.65:34791 every 8 connection(s)
I20260929 18:32:25.319556 1192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
I20260929 18:32:25.321097 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 1192
I20260929 18:32:25.321301 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance
I20260929 18:32:25.321707 29501 external_mini_cluster.cc:1428] Setting key 78d701cfbb822156a8d461fad7eb6fec
Sep 29 18:32:25 dist-test-slave-qcpv krb5kdc[1086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706745, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.207.66@KRBTEST.COM; defaulting to no policy
I20260929 18:32:25.336153 1322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:25.322122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:54093 (local address 127.28.207.126:40703)
0929 18:32:25.322513 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:25.322519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:25.323750 (+ 1231us) server_negotiation.cc:407] Connection header received
0929 18:32:25.325339 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:25.325343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:25.325403 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:25.325505 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:25.326907 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:25.327491 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:25.328776 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:25.329018 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:25.329153 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:25.332032 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:25.332035 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:25.332038 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:25.332064 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:25.333823 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:25.334433 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:25.334437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:25.334439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:25.334528 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:25.334926 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:25.334929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:25.334931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:25.335145 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:25.335260 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:25.335818 (+ 558us) server_negotiation.cc:299] Negotiation successful
0929 18:32:25.335957 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":111,"threads_started":1}
I20260929 18:32:25.337090 1323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:25.322730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:40703 (local address 127.28.207.65:54093)
0929 18:32:25.323537 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:25.323617 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:25.325074 (+ 1457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:25.325697 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:25.325705 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:25.326186 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:25.326725 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:25.326738 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:25.327692 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:25.327696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:25.328632 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:25.328641 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:25.329201 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:25.329206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:25.329405 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:25.330101 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:25.330118 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:25.331810 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:25.333974 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:25.333981 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:25.334001 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:25.334315 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:25.334641 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:25.334644 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:25.334646 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:25.334799 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:25.335257 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:25.335265 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:25.335563 (+ 298us) client_negotiation.cc:769] Sending connection context
0929 18:32:25.335779 (+ 216us) client_negotiation.cc:240] Negotiation successful
0929 18:32:25.336136 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":175,"threads_started":1}
I20260929 18:32:25.338696 1320 heartbeater.cc:347] Connected to a master server at 127.28.207.126:40703
I20260929 18:32:25.339001 1320 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:25.339485 1320 heartbeater.cc:511] Master 127.28.207.126:40703 requested a full tablet report, sending...
I20260929 18:32:25.341264 1130 ts_manager.cc:194] Registered new tserver with Master: 7656fb48bc0f4b68bf088a82abff9728 (127.28.207.65:34791)
I20260929 18:32:25.342434 1130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.65@KRBTEST.COM'} at 127.28.207.65:54093
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab.
Entry for principal kudu/127.28.207.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab.
W20260929 18:32:25.514410 29501 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.28.207.66: real 0.190s user 0.000s sys 0.004s
I20260929 18:32:25.515491 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.66:0
--local_ip_for_outbound_sockets=127.28.207.66
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:40703
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:36699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:25.629833 1326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:25.630112 1326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:25.630158 1326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:25.634254 1326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:25.634335 1326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:25.634397 1326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.66
I20260929 18:32:25.639765 1326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:36699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:40703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.66
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:25.640868 1326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:25.641921 1326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:25.653255 1332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:25.653337 1331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:25.653508 1326 server_base.cc:1034] running on GCE node
W20260929 18:32:25.653714 1334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:25.654420 1326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:25.655447 1326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:25.656687 1326 hybrid_clock.cc:648] HybridClock initialized: now 1790706745656648 us; error 54 us; skew 500 ppm
Sep 29 18:32:25 dist-test-slave-qcpv krb5kdc[1086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706745, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:25.660717 1326 init.cc:377] Logged in from keytab as kudu/127.28.207.66@KRBTEST.COM (short username kudu)
I20260929 18:32:25.661772 1326 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:25.661886 1326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:25.662189 1326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:25.664925 1326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-1/data/instance:
uuid: "215cca5e6a2d4d1aa326af4cd843fd48"
format_stamp: "Formatted at 2026-09-29 18:32:25 on dist-test-slave-qcpv"
server_key: "d6e5b20877dd675a6b2a208974a0e223"
server_key_iv: "2007161eea65b9804e44a2a49b8e50c0"
server_key_version: "encryptionkey@0"
I20260929 18:32:25.665629 1326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance:
uuid: "215cca5e6a2d4d1aa326af4cd843fd48"
format_stamp: "Formatted at 2026-09-29 18:32:25 on dist-test-slave-qcpv"
server_key: "d6e5b20877dd675a6b2a208974a0e223"
server_key_iv: "2007161eea65b9804e44a2a49b8e50c0"
server_key_version: "encryptionkey@0"
I20260929 18:32:25.670548 1326 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 18:32:25.674118 1340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:25.676118 1326 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:32:25.676354 1326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-1/wal
uuid: "215cca5e6a2d4d1aa326af4cd843fd48"
format_stamp: "Formatted at 2026-09-29 18:32:25 on dist-test-slave-qcpv"
server_key: "d6e5b20877dd675a6b2a208974a0e223"
server_key_iv: "2007161eea65b9804e44a2a49b8e50c0"
server_key_version: "encryptionkey@0"
I20260929 18:32:25.676522 1326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:25.706508 1326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:25.711324 1326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:25.711552 1326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:25.712302 1326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:32:25.713454 1326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:25.713505 1326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:25.713588 1326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:25.713627 1326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:25.725623 1326 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.66:38669
I20260929 18:32:25.725649 1453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.66:38669 every 8 connection(s)
I20260929 18:32:25.726486 1326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
I20260929 18:32:25.734947 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 1326
I20260929 18:32:25.735110 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance
I20260929 18:32:25.735459 29501 external_mini_cluster.cc:1428] Setting key fccf98225df74d7041000aa35e8ac809
Sep 29 18:32:25 dist-test-slave-qcpv krb5kdc[1086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706745, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:32:25.743709 1322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:25.728758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:54045 (local address 127.28.207.126:40703)
0929 18:32:25.729026 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:25.729031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:25.729812 (+ 781us) server_negotiation.cc:407] Connection header received
0929 18:32:25.731543 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:25.731546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:25.731604 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:25.731688 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:25.733383 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:25.733958 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:25.736804 (+ 2846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:25.737048 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:25.737181 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:25.739980 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:25.739984 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:25.739986 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:25.740009 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:25.741489 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:25.742044 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:25.742052 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:25.742054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:25.742128 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:25.742495 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:25.742498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:25.742500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:25.742699 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:25.742806 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:25.743399 (+ 593us) server_negotiation.cc:299] Negotiation successful
0929 18:32:25.743542 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260929 18:32:25.744582 1456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:25.729077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:40703 (local address 127.28.207.66:54045)
0929 18:32:25.729633 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:25.729682 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:25.731131 (+ 1449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:25.731947 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:25.731957 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:25.732482 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:25.733145 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:25.733163 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:25.734149 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:25.734153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:25.736636 (+ 2483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:25.736657 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:25.737254 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:25.737259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:25.737470 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:25.738174 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:25.738192 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:25.739774 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:25.741622 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:25.741628 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:25.741646 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:25.741929 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:25.742220 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:25.742225 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:25.742227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:25.742386 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:25.742814 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:25.742824 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:25.743147 (+ 323us) client_negotiation.cc:769] Sending connection context
0929 18:32:25.743351 (+ 204us) client_negotiation.cc:240] Negotiation successful
0929 18:32:25.743707 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":148,"threads_started":1}
I20260929 18:32:25.746466 1454 heartbeater.cc:347] Connected to a master server at 127.28.207.126:40703
I20260929 18:32:25.746766 1454 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.28.207.67@KRBTEST.COM; defaulting to no policy
I20260929 18:32:25.747381 1454 heartbeater.cc:511] Master 127.28.207.126:40703 requested a full tablet report, sending...
I20260929 18:32:25.748646 1130 ts_manager.cc:194] Registered new tserver with Master: 215cca5e6a2d4d1aa326af4cd843fd48 (127.28.207.66:38669)
I20260929 18:32:25.749289 1130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.66@KRBTEST.COM'} at 127.28.207.66:54045
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab.
Entry for principal kudu/127.28.207.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab.
I20260929 18:32:25.772944 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.67:0
--local_ip_for_outbound_sockets=127.28.207.67
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:40703
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:36699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:25.898669 1459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:25.899073 1459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:25.899173 1459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:25.903784 1459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:25.903924 1459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:25.904033 1459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.67
I20260929 18:32:25.909791 1459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:36699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:40703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.67
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:25.911068 1459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:25.912153 1459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:25.923111 1464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:25.923131 1467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:25.923131 1465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:25.923616 1459 server_base.cc:1034] running on GCE node
I20260929 18:32:25.924256 1459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:25.925222 1459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:25.926538 1459 hybrid_clock.cc:648] HybridClock initialized: now 1790706745926479 us; error 77 us; skew 500 ppm
Sep 29 18:32:25 dist-test-slave-qcpv krb5kdc[1086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706745, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:25.930413 1459 init.cc:377] Logged in from keytab as kudu/127.28.207.67@KRBTEST.COM (short username kudu)
I20260929 18:32:25.931535 1459 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:25.931617 1459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:25.932021 1459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:25.934401 1459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-2/data/instance:
uuid: "44590e8af00e4d909dd1610f9f4a4bb3"
format_stamp: "Formatted at 2026-09-29 18:32:25 on dist-test-slave-qcpv"
server_key: "21cb8e3d80344e67b76d7ce919dd896d"
server_key_iv: "dfeca9e6a3f67e9eb51cdb98cc746e45"
server_key_version: "encryptionkey@0"
I20260929 18:32:25.935390 1459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance:
uuid: "44590e8af00e4d909dd1610f9f4a4bb3"
format_stamp: "Formatted at 2026-09-29 18:32:25 on dist-test-slave-qcpv"
server_key: "21cb8e3d80344e67b76d7ce919dd896d"
server_key_iv: "dfeca9e6a3f67e9eb51cdb98cc746e45"
server_key_version: "encryptionkey@0"
I20260929 18:32:25.940901 1459 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 18:32:25.944423 1473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:25.946082 1459 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:32:25.946223 1459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-2/wal
uuid: "44590e8af00e4d909dd1610f9f4a4bb3"
format_stamp: "Formatted at 2026-09-29 18:32:25 on dist-test-slave-qcpv"
server_key: "21cb8e3d80344e67b76d7ce919dd896d"
server_key_iv: "dfeca9e6a3f67e9eb51cdb98cc746e45"
server_key_version: "encryptionkey@0"
I20260929 18:32:25.946425 1459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:25.972800 1459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:25.978844 1459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:25.979350 1459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:25.980240 1459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:32:25.981539 1459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:25.981595 1459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:25.981673 1459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:25.981707 1459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:25.995790 1459 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.67:45613
I20260929 18:32:25.995831 1586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.67:45613 every 8 connection(s)
I20260929 18:32:25.996893 1459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
I20260929 18:32:26.003109 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 1459
I20260929 18:32:26.003316 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance
I20260929 18:32:26.003749 29501 external_mini_cluster.cc:1428] Setting key 0be1a417aa1e644d9d4756c333f7a347
Sep 29 18:32:26 dist-test-slave-qcpv krb5kdc[1086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706745, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:32:26.016160 1322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:25.999260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:52127 (local address 127.28.207.126:40703)
0929 18:32:25.999482 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:25.999488 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:26.000583 (+ 1095us) server_negotiation.cc:407] Connection header received
0929 18:32:26.002301 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:26.002306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:26.002377 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:26.002479 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:26.004458 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:26.005264 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:26.007137 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:26.007396 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:26.007501 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:26.011353 (+ 3852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:26.011358 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:26.011360 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:26.011398 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:26.013033 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:26.013987 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:26.013990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:26.013991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:26.014076 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:26.014513 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:26.014516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:26.014517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:26.014723 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:26.014811 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:26.015900 (+ 1089us) server_negotiation.cc:299] Negotiation successful
0929 18:32:26.016025 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260929 18:32:26.017782 1589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:25.999666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:40703 (local address 127.28.207.67:52127)
0929 18:32:26.000291 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:26.000355 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:26.001979 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:26.002713 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:26.002721 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:26.003453 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:26.004248 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:26.004272 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:26.005511 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:26.005519 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:26.006985 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:26.007001 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:26.007648 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:26.007653 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:26.007897 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:26.008958 (+ 1061us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:26.008980 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:26.011011 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:26.013215 (+ 2204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:26.013222 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:26.013243 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:26.013825 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:26.014205 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:26.014208 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:26.014210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:26.014404 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:26.014847 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:26.014931 (+ 84us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:26.015578 (+ 647us) client_negotiation.cc:769] Sending connection context
0929 18:32:26.015802 (+ 224us) client_negotiation.cc:240] Negotiation successful
0929 18:32:26.016156 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":132,"threads_started":1}
I20260929 18:32:26.020231 1587 heartbeater.cc:347] Connected to a master server at 127.28.207.126:40703
I20260929 18:32:26.020489 1587 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:26.020954 1587 heartbeater.cc:511] Master 127.28.207.126:40703 requested a full tablet report, sending...
I20260929 18:32:26.022125 1130 ts_manager.cc:194] Registered new tserver with Master: 44590e8af00e4d909dd1610f9f4a4bb3 (127.28.207.67:45613)
I20260929 18:32:26.022727 1130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.67@KRBTEST.COM'} at 127.28.207.67:52127
I20260929 18:32:26.033607 29501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:32:26.047464 1322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:26.037460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54162 (local address 127.28.207.126:40703)
0929 18:32:26.037676 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:26.037683 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:26.037854 (+ 171us) server_negotiation.cc:407] Connection header received
0929 18:32:26.038035 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:26.038038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:26.038094 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:26.038191 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:26.039033 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:26.039583 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:26.040579 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:26.040818 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:26.040932 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:26.042512 (+ 1580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:26.042516 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:26.042518 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:26.042547 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:26.044877 (+ 2330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:26.045684 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:26.045690 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:26.045696 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:26.045801 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:26.046258 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:26.046260 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:26.046262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:26.046519 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:26.046681 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:26.047006 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 18:32:26.047137 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 18:32:26.052883 1130 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:32:26.057695 1130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:32:26.078437 1599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:26.070992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:38669 (local address 127.0.0.1:37578)
0929 18:32:26.071844 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:26.071905 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:26.072059 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:26.073144 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:26.073147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:26.073168 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:26.073329 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:26.073337 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:26.074769 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:26.074773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:26.076223 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:26.076235 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:26.077630 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:26.077634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:26.077905 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:26.078022 (+ 117us) client_negotiation.cc:769] Sending connection context
0929 18:32:26.078102 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 18:32:26.078272 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":177,"thread_start_us":325,"threads_started":1}
I20260929 18:32:26.078445 1602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:26.071571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50248 (local address 127.28.207.65:34791)
0929 18:32:26.072205 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:26.072216 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:26.072246 (+ 30us) server_negotiation.cc:407] Connection header received
0929 18:32:26.072345 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:26.072350 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:26.072552 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:26.073193 (+ 641us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:26.073745 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:26.074844 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:26.076314 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:26.077246 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:26.077352 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:26.077542 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:26.078170 (+ 628us) server_negotiation.cc:299] Negotiation successful
0929 18:32:26.078281 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":132,"threads_started":1}
I20260929 18:32:26.078545 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:26.071103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37578 (local address 127.28.207.66:38669)
0929 18:32:26.072369 (+ 1266us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:26.072388 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:26.072413 (+ 25us) server_negotiation.cc:407] Connection header received
0929 18:32:26.072498 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:26.072503 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:26.072739 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:26.072939 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:26.073466 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:26.074614 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:26.076401 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:26.077460 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:26.077779 (+ 319us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:26.078120 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:26.078199 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 18:32:26.078321 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1165,"thread_start_us":115,"threads_started":1}
I20260929 18:32:26.079020 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:26.072904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:45613 (local address 127.0.0.1:54416)
0929 18:32:26.073437 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:26.073452 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:26.073544 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:26.074195 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:26.074199 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:26.074217 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:26.074304 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:26.074310 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:26.075698 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:26.075702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:26.076844 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:26.076854 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:26.078189 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:26.078193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:26.078282 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:26.078306 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 18:32:26.078379 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 18:32:26.078465 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":128,"threads_started":1}
I20260929 18:32:26.079033 1601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:26.071372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:34791 (local address 127.0.0.1:50248)
0929 18:32:26.072026 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:26.072046 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:26.072172 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:26.072720 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:26.072727 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:26.072754 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:26.072952 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:26.073629 (+ 677us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:26.074987 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:26.074991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:26.076144 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:26.076154 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:26.077605 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:26.077611 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:26.077818 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:26.077983 (+ 165us) client_negotiation.cc:769] Sending connection context
0929 18:32:26.078746 (+ 763us) client_negotiation.cc:240] Negotiation successful
0929 18:32:26.078876 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"mutex_wait_us":53,"thread_start_us":161,"threads_started":1}
I20260929 18:32:26.079471 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:26.073098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54416 (local address 127.28.207.67:45613)
0929 18:32:26.073612 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:26.073623 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:26.073644 (+ 21us) server_negotiation.cc:407] Connection header received
0929 18:32:26.073724 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:26.073729 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:26.073905 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:26.074048 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:26.074420 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:26.075537 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:26.077022 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:26.077870 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:26.077996 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:26.078124 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:26.079160 (+ 1036us) server_negotiation.cc:299] Negotiation successful
0929 18:32:26.079320 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":180,"threads_started":1}
I20260929 18:32:26.082558 1521 tablet_service.cc:1467] Processing CreateTablet for tablet d36f265444044dedbea41cc6b449c455 (DEFAULT_TABLE table=test-table [id=3a0524e233294ff4b1a4269c9890742a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:26.082731 1254 tablet_service.cc:1467] Processing CreateTablet for tablet d36f265444044dedbea41cc6b449c455 (DEFAULT_TABLE table=test-table [id=3a0524e233294ff4b1a4269c9890742a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:26.082746 1388 tablet_service.cc:1467] Processing CreateTablet for tablet d36f265444044dedbea41cc6b449c455 (DEFAULT_TABLE table=test-table [id=3a0524e233294ff4b1a4269c9890742a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:26.083853 1521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d36f265444044dedbea41cc6b449c455. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:26.084090 1254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d36f265444044dedbea41cc6b449c455. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:26.084090 1388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d36f265444044dedbea41cc6b449c455. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:26.091138 1605 tablet_bootstrap.cc:492] T d36f265444044dedbea41cc6b449c455 P 44590e8af00e4d909dd1610f9f4a4bb3: Bootstrap starting.
I20260929 18:32:26.093644 1607 tablet_bootstrap.cc:492] T d36f265444044dedbea41cc6b449c455 P 215cca5e6a2d4d1aa326af4cd843fd48: Bootstrap starting.
I20260929 18:32:26.093644 1606 tablet_bootstrap.cc:492] T d36f265444044dedbea41cc6b449c455 P 7656fb48bc0f4b68bf088a82abff9728: Bootstrap starting.
I20260929 18:32:26.094484 1605 tablet_bootstrap.cc:654] T d36f265444044dedbea41cc6b449c455 P 44590e8af00e4d909dd1610f9f4a4bb3: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:26.095687 1605 log.cc:824] T d36f265444044dedbea41cc6b449c455 P 44590e8af00e4d909dd1610f9f4a4bb3: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:26.096338 1607 tablet_bootstrap.cc:654] T d36f265444044dedbea41cc6b449c455 P 215cca5e6a2d4d1aa326af4cd843fd48: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:26.096397 1606 tablet_bootstrap.cc:654] T d36f265444044dedbea41cc6b449c455 P 7656fb48bc0f4b68bf088a82abff9728: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:26.097512 1606 log.cc:824] T d36f265444044dedbea41cc6b449c455 P 7656fb48bc0f4b68bf088a82abff9728: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:26.097733 1607 log.cc:824] T d36f265444044dedbea41cc6b449c455 P 215cca5e6a2d4d1aa326af4cd843fd48: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:26.097993 1605 tablet_bootstrap.cc:492] T d36f265444044dedbea41cc6b449c455 P 44590e8af00e4d909dd1610f9f4a4bb3: No bootstrap required, opened a new log
I20260929 18:32:26.098203 1605 ts_tablet_manager.cc:1397] T d36f265444044dedbea41cc6b449c455 P 44590e8af00e4d909dd1610f9f4a4bb3: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 18:32:26.100069 1607 tablet_bootstrap.cc:492] T d36f265444044dedbea41cc6b449c455 P 215cca5e6a2d4d1aa326af4cd843fd48: No bootstrap required, opened a new log
I20260929 18:32:26.100069 1606 tablet_bootstrap.cc:492] T d36f265444044dedbea41cc6b449c455 P 7656fb48bc0f4b68bf088a82abff9728: No bootstrap required, opened a new log
I20260929 18:32:26.100282 1606 ts_tablet_manager.cc:1397] T d36f265444044dedbea41cc6b449c455 P 7656fb48bc0f4b68bf088a82abff9728: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 18:32:26.100282 1607 ts_tablet_manager.cc:1397] T d36f265444044dedbea41cc6b449c455 P 215cca5e6a2d4d1aa326af4cd843fd48: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260929 18:32:26.101497 1605 raft_consensus.cc:348] T d36f265444044dedbea41cc6b449c455 P 44590e8af00e4d909dd1610f9f4a4bb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215cca5e6a2d4d1aa326af4cd843fd48" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38669 } } peers { permanent_uuid: "7656fb48bc0f4b68bf088a82abff9728" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 34791 } } peers { permanent_uuid: "44590e8af00e4d909dd1610f9f4a4bb3" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 45613 } }
I20260929 18:32:26.101722 1605 raft_consensus.cc:374] T d36f265444044dedbea41cc6b449c455 P 44590e8af00e4d909dd1610f9f4a4bb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:26.101759 1605 raft_consensus.cc:729] T d36f265444044dedbea41cc6b449c455 P 44590e8af00e4d909dd1610f9f4a4bb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44590e8af00e4d909dd1610f9f4a4bb3, State: Initialized, Role: FOLLOWER
I20260929 18:32:26.102182 1605 consensus_queue.cc:260] T d36f265444044dedbea41cc6b449c455 P 44590e8af00e4d909dd1610f9f4a4bb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215cca5e6a2d4d1aa326af4cd843fd48" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38669 } } peers { permanent_uuid: "7656fb48bc0f4b68bf088a82abff9728" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 34791 } } peers { permanent_uuid: "44590e8af00e4d909dd1610f9f4a4bb3" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 45613 } }
I20260929 18:32:26.102819 1587 heartbeater.cc:503] Master 127.28.207.126:40703 was elected leader, sending a full tablet report...
I20260929 18:32:26.103278 1605 ts_tablet_manager.cc:1428] T d36f265444044dedbea41cc6b449c455 P 44590e8af00e4d909dd1610f9f4a4bb3: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 18:32:26.105185 1606 raft_consensus.cc:348] T d36f265444044dedbea41cc6b449c455 P 7656fb48bc0f4b68bf088a82abff9728 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215cca5e6a2d4d1aa326af4cd843fd48" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38669 } } peers { permanent_uuid: "7656fb48bc0f4b68bf088a82abff9728" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 34791 } } peers { permanent_uuid: "44590e8af00e4d909dd1610f9f4a4bb3" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 45613 } }
I20260929 18:32:26.105083 1607 raft_consensus.cc:348] T d36f265444044dedbea41cc6b449c455 P 215cca5e6a2d4d1aa326af4cd843fd48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215cca5e6a2d4d1aa326af4cd843fd48" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38669 } } peers { permanent_uuid: "7656fb48bc0f4b68bf088a82abff9728" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 34791 } } peers { permanent_uuid: "44590e8af00e4d909dd1610f9f4a4bb3" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 45613 } }
I20260929 18:32:26.105464 1607 raft_consensus.cc:374] T d36f265444044dedbea41cc6b449c455 P 215cca5e6a2d4d1aa326af4cd843fd48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:26.105464 1606 raft_consensus.cc:374] T d36f265444044dedbea41cc6b449c455 P 7656fb48bc0f4b68bf088a82abff9728 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:26.105520 1606 raft_consensus.cc:729] T d36f265444044dedbea41cc6b449c455 P 7656fb48bc0f4b68bf088a82abff9728 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7656fb48bc0f4b68bf088a82abff9728, State: Initialized, Role: FOLLOWER
I20260929 18:32:26.105528 1607 raft_consensus.cc:729] T d36f265444044dedbea41cc6b449c455 P 215cca5e6a2d4d1aa326af4cd843fd48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 215cca5e6a2d4d1aa326af4cd843fd48, State: Initialized, Role: FOLLOWER
I20260929 18:32:26.106226 1607 consensus_queue.cc:260] T d36f265444044dedbea41cc6b449c455 P 215cca5e6a2d4d1aa326af4cd843fd48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215cca5e6a2d4d1aa326af4cd843fd48" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38669 } } peers { permanent_uuid: "7656fb48bc0f4b68bf088a82abff9728" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 34791 } } peers { permanent_uuid: "44590e8af00e4d909dd1610f9f4a4bb3" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 45613 } }
I20260929 18:32:26.106226 1606 consensus_queue.cc:260] T d36f265444044dedbea41cc6b449c455 P 7656fb48bc0f4b68bf088a82abff9728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215cca5e6a2d4d1aa326af4cd843fd48" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38669 } } peers { permanent_uuid: "7656fb48bc0f4b68bf088a82abff9728" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 34791 } } peers { permanent_uuid: "44590e8af00e4d909dd1610f9f4a4bb3" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 45613 } }
I20260929 18:32:26.107425 1320 heartbeater.cc:503] Master 127.28.207.126:40703 was elected leader, sending a full tablet report...
I20260929 18:32:26.107921 1606 ts_tablet_manager.cc:1428] T d36f265444044dedbea41cc6b449c455 P 7656fb48bc0f4b68bf088a82abff9728: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260929 18:32:26.107921 1607 ts_tablet_manager.cc:1428] T d36f265444044dedbea41cc6b449c455 P 215cca5e6a2d4d1aa326af4cd843fd48: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260929 18:32:26.108690 1454 heartbeater.cc:503] Master 127.28.207.126:40703 was elected leader, sending a full tablet report...
I20260929 18:32:26.186060 1611 raft_consensus.cc:482] T d36f265444044dedbea41cc6b449c455 P 44590e8af00e4d909dd1610f9f4a4bb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:32:26.186353 1611 raft_consensus.cc:504] T d36f265444044dedbea41cc6b449c455 P 44590e8af00e4d909dd1610f9f4a4bb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215cca5e6a2d4d1aa326af4cd843fd48" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38669 } } peers { permanent_uuid: "7656fb48bc0f4b68bf088a82abff9728" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 34791 } } peers { permanent_uuid: "44590e8af00e4d909dd1610f9f4a4bb3" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 45613 } }
I20260929 18:32:26.187650 1611 leader_election.cc:288] T d36f265444044dedbea41cc6b449c455 P 44590e8af00e4d909dd1610f9f4a4bb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 215cca5e6a2d4d1aa326af4cd843fd48 (127.28.207.66:38669), 7656fb48bc0f4b68bf088a82abff9728 (127.28.207.65:34791)
I20260929 18:32:26.193921 1602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:26.188501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:41031 (local address 127.28.207.65:34791)
0929 18:32:26.188667 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:26.188672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:26.188726 (+ 54us) server_negotiation.cc:407] Connection header received
0929 18:32:26.188945 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:26.188950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:26.189013 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:26.189125 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:26.189443 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:26.190190 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:26.192135 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:26.192825 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:26.192974 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:26.193043 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:26.193646 (+ 603us) server_negotiation.cc:299] Negotiation successful
0929 18:32:26.193774 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 18:32:26.193921 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:26.188216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:53295 (local address 127.28.207.66:38669)
0929 18:32:26.188363 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:26.188369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:26.188552 (+ 183us) server_negotiation.cc:407] Connection header received
0929 18:32:26.188653 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:26.188658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:26.188739 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:26.188957 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:26.189125 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:26.190165 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:26.192136 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:26.192825 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:26.192974 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:26.193043 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:26.193641 (+ 598us) server_negotiation.cc:299] Negotiation successful
0929 18:32:26.193774 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 18:32:26.193919 1589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:26.188121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:38669 (local address 127.28.207.67:53295)
0929 18:32:26.188365 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:26.188384 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:26.188532 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:26.188912 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:26.188915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:26.188928 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:26.189034 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:26.189039 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:26.190324 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:26.190328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:26.191951 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:26.191962 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:26.193144 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:26.193149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:26.193430 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:26.193471 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 18:32:26.193569 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 18:32:26.193700 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":307}
I20260929 18:32:26.193934 1614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:26.188121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:34791 (local address 127.28.207.67:41031)
0929 18:32:26.188647 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:26.188661 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:26.188746 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:26.189154 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:26.189158 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:26.189179 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:26.189262 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:26.189266 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:26.190324 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:26.190328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:26.191951 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:26.191962 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:26.193144 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:26.193149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:26.193424 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:26.193481 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 18:32:26.193569 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 18:32:26.193700 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":312,"thread_start_us":182,"threads_started":1}
I20260929 18:32:26.194835 1408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d36f265444044dedbea41cc6b449c455" candidate_uuid: "44590e8af00e4d909dd1610f9f4a4bb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "215cca5e6a2d4d1aa326af4cd843fd48" is_pre_election: true
I20260929 18:32:26.195274 1408 raft_consensus.cc:2454] T d36f265444044dedbea41cc6b449c455 P 215cca5e6a2d4d1aa326af4cd843fd48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44590e8af00e4d909dd1610f9f4a4bb3 in term 0.
I20260929 18:32:26.195242 1274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d36f265444044dedbea41cc6b449c455" candidate_uuid: "44590e8af00e4d909dd1610f9f4a4bb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7656fb48bc0f4b68bf088a82abff9728" is_pre_election: true
I20260929 18:32:26.195582 1274 raft_consensus.cc:2454] T d36f265444044dedbea41cc6b449c455 P 7656fb48bc0f4b68bf088a82abff9728 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44590e8af00e4d909dd1610f9f4a4bb3 in term 0.
I20260929 18:32:26.195827 1476 leader_election.cc:302] T d36f265444044dedbea41cc6b449c455 P 44590e8af00e4d909dd1610f9f4a4bb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 215cca5e6a2d4d1aa326af4cd843fd48, 44590e8af00e4d909dd1610f9f4a4bb3; no voters:
I20260929 18:32:26.196170 1611 raft_consensus.cc:2781] T d36f265444044dedbea41cc6b449c455 P 44590e8af00e4d909dd1610f9f4a4bb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:32:26.196262 1611 raft_consensus.cc:482] T d36f265444044dedbea41cc6b449c455 P 44590e8af00e4d909dd1610f9f4a4bb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:32:26.196305 1611 raft_consensus.cc:3037] T d36f265444044dedbea41cc6b449c455 P 44590e8af00e4d909dd1610f9f4a4bb3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:26.197770 1611 raft_consensus.cc:504] T d36f265444044dedbea41cc6b449c455 P 44590e8af00e4d909dd1610f9f4a4bb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215cca5e6a2d4d1aa326af4cd843fd48" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38669 } } peers { permanent_uuid: "7656fb48bc0f4b68bf088a82abff9728" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 34791 } } peers { permanent_uuid: "44590e8af00e4d909dd1610f9f4a4bb3" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 45613 } }
I20260929 18:32:26.198254 1611 leader_election.cc:288] T d36f265444044dedbea41cc6b449c455 P 44590e8af00e4d909dd1610f9f4a4bb3 [CANDIDATE]: Term 1 election: Requested vote from peers 215cca5e6a2d4d1aa326af4cd843fd48 (127.28.207.66:38669), 7656fb48bc0f4b68bf088a82abff9728 (127.28.207.65:34791)
I20260929 18:32:26.198657 1274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d36f265444044dedbea41cc6b449c455" candidate_uuid: "44590e8af00e4d909dd1610f9f4a4bb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7656fb48bc0f4b68bf088a82abff9728"
I20260929 18:32:26.198729 1408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d36f265444044dedbea41cc6b449c455" candidate_uuid: "44590e8af00e4d909dd1610f9f4a4bb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "215cca5e6a2d4d1aa326af4cd843fd48"
I20260929 18:32:26.198825 1274 raft_consensus.cc:3037] T d36f265444044dedbea41cc6b449c455 P 7656fb48bc0f4b68bf088a82abff9728 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:26.199008 1408 raft_consensus.cc:3037] T d36f265444044dedbea41cc6b449c455 P 215cca5e6a2d4d1aa326af4cd843fd48 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:26.200028 1274 raft_consensus.cc:2454] T d36f265444044dedbea41cc6b449c455 P 7656fb48bc0f4b68bf088a82abff9728 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44590e8af00e4d909dd1610f9f4a4bb3 in term 1.
I20260929 18:32:26.200414 1477 leader_election.cc:302] T d36f265444044dedbea41cc6b449c455 P 44590e8af00e4d909dd1610f9f4a4bb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44590e8af00e4d909dd1610f9f4a4bb3, 7656fb48bc0f4b68bf088a82abff9728; no voters:
I20260929 18:32:26.200593 1408 raft_consensus.cc:2454] T d36f265444044dedbea41cc6b449c455 P 215cca5e6a2d4d1aa326af4cd843fd48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44590e8af00e4d909dd1610f9f4a4bb3 in term 1.
I20260929 18:32:26.200730 1611 raft_consensus.cc:2781] T d36f265444044dedbea41cc6b449c455 P 44590e8af00e4d909dd1610f9f4a4bb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:32:26.200975 1611 raft_consensus.cc:686] T d36f265444044dedbea41cc6b449c455 P 44590e8af00e4d909dd1610f9f4a4bb3 [term 1 LEADER]: Becoming Leader. State: Replica: 44590e8af00e4d909dd1610f9f4a4bb3, State: Running, Role: LEADER
I20260929 18:32:26.201323 1611 consensus_queue.cc:237] T d36f265444044dedbea41cc6b449c455 P 44590e8af00e4d909dd1610f9f4a4bb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215cca5e6a2d4d1aa326af4cd843fd48" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38669 } } peers { permanent_uuid: "7656fb48bc0f4b68bf088a82abff9728" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 34791 } } peers { permanent_uuid: "44590e8af00e4d909dd1610f9f4a4bb3" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 45613 } }
I20260929 18:32:26.204499 1129 catalog_manager.cc:5495] T d36f265444044dedbea41cc6b449c455 P 44590e8af00e4d909dd1610f9f4a4bb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44590e8af00e4d909dd1610f9f4a4bb3 (127.28.207.67). New cstate: current_term: 1 leader_uuid: "44590e8af00e4d909dd1610f9f4a4bb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "215cca5e6a2d4d1aa326af4cd843fd48" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7656fb48bc0f4b68bf088a82abff9728" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 34791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44590e8af00e4d909dd1610f9f4a4bb3" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 45613 } health_report { overall_health: HEALTHY } } }
W20260929 18:32:26.227790 1455 tablet.cc:2369] T d36f265444044dedbea41cc6b449c455 P 215cca5e6a2d4d1aa326af4cd843fd48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:32:26.235246 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:26.230733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54430 (local address 127.28.207.67:45613)
0929 18:32:26.230887 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:26.230891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:26.230904 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:32:26.231234 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:26.231236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:26.231295 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:26.231423 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:32:26.231737 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:26.232253 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:26.233323 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:26.233557 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:26.233652 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:26.234094 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:32:26.234675 (+ 581us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:32:26.234779 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:26.235036 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 18:32:26.235098 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:32:26.242484 1274 raft_consensus.cc:1261] T d36f265444044dedbea41cc6b449c455 P 7656fb48bc0f4b68bf088a82abff9728 [term 1 FOLLOWER]: Refusing update from remote peer 44590e8af00e4d909dd1610f9f4a4bb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:32:26.242530 1408 raft_consensus.cc:1261] T d36f265444044dedbea41cc6b449c455 P 215cca5e6a2d4d1aa326af4cd843fd48 [term 1 FOLLOWER]: Refusing update from remote peer 44590e8af00e4d909dd1610f9f4a4bb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:32:26.243184 1611 consensus_queue.cc:1035] T d36f265444044dedbea41cc6b449c455 P 44590e8af00e4d909dd1610f9f4a4bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7656fb48bc0f4b68bf088a82abff9728" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 34791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:32:26.243381 1615 consensus_queue.cc:1035] T d36f265444044dedbea41cc6b449c455 P 44590e8af00e4d909dd1610f9f4a4bb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "215cca5e6a2d4d1aa326af4cd843fd48" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 18:32:26.247980 1588 tablet.cc:2369] T d36f265444044dedbea41cc6b449c455 P 44590e8af00e4d909dd1610f9f4a4bb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:32:26.254194 1621 mvcc.cc:205] Tried to move back new op lower bound from 7334734832601825280 to 7334734832444116992. Current Snapshot: MvccSnapshot[applied={T|T < 7334734832601825280}]
I20260929 18:32:26.255106 1619 mvcc.cc:205] Tried to move back new op lower bound from 7334734832601825280 to 7334734832444116992. Current Snapshot: MvccSnapshot[applied={T|T < 7334734832601825280}]
I20260929 18:32:26.262960 1130 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:32:26.263207 1130 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:32:26.266377 1130 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1fed4d7ba6804091a3941424286bc421: Sending DeleteTablet for 3 replicas of tablet d36f265444044dedbea41cc6b449c455
I20260929 18:32:26.267637 1254 tablet_service.cc:1514] Processing DeleteTablet for tablet d36f265444044dedbea41cc6b449c455 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:26 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:50248
I20260929 18:32:26.267728 1521 tablet_service.cc:1514] Processing DeleteTablet for tablet d36f265444044dedbea41cc6b449c455 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:26 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:54416
I20260929 18:32:26.268347 1388 tablet_service.cc:1514] Processing DeleteTablet for tablet d36f265444044dedbea41cc6b449c455 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:26 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:37578
I20260929 18:32:26.268379 1628 tablet_replica.cc:330] T d36f265444044dedbea41cc6b449c455 P 7656fb48bc0f4b68bf088a82abff9728: stopping tablet replica
I20260929 18:32:26.268805 1628 raft_consensus.cc:2229] T d36f265444044dedbea41cc6b449c455 P 7656fb48bc0f4b68bf088a82abff9728 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:32:26.268957 1630 tablet_replica.cc:330] T d36f265444044dedbea41cc6b449c455 P 215cca5e6a2d4d1aa326af4cd843fd48: stopping tablet replica
I20260929 18:32:26.269186 1628 raft_consensus.cc:2258] T d36f265444044dedbea41cc6b449c455 P 7656fb48bc0f4b68bf088a82abff9728 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:26.269465 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 1192
I20260929 18:32:26.269474 1630 raft_consensus.cc:2229] T d36f265444044dedbea41cc6b449c455 P 215cca5e6a2d4d1aa326af4cd843fd48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:32:26.269960 1630 raft_consensus.cc:2258] T d36f265444044dedbea41cc6b449c455 P 215cca5e6a2d4d1aa326af4cd843fd48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:26.270269 1629 tablet_replica.cc:330] T d36f265444044dedbea41cc6b449c455 P 44590e8af00e4d909dd1610f9f4a4bb3: stopping tablet replica
I20260929 18:32:26.270586 1629 raft_consensus.cc:2229] T d36f265444044dedbea41cc6b449c455 P 44590e8af00e4d909dd1610f9f4a4bb3 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:32:26.271001 1629 raft_consensus.cc:2258] T d36f265444044dedbea41cc6b449c455 P 44590e8af00e4d909dd1610f9f4a4bb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:26.272231 1630 ts_tablet_manager.cc:1905] T d36f265444044dedbea41cc6b449c455 P 215cca5e6a2d4d1aa326af4cd843fd48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:32:26.272859 1629 ts_tablet_manager.cc:1905] T d36f265444044dedbea41cc6b449c455 P 44590e8af00e4d909dd1610f9f4a4bb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:32:26.276089 1629 ts_tablet_manager.cc:1918] T d36f265444044dedbea41cc6b449c455 P 44590e8af00e4d909dd1610f9f4a4bb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:32:26.276203 1629 log.cc:1196] T d36f265444044dedbea41cc6b449c455 P 44590e8af00e4d909dd1610f9f4a4bb3: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-2/wal/wals/d36f265444044dedbea41cc6b449c455
I20260929 18:32:26.276532 1629 ts_tablet_manager.cc:1939] T d36f265444044dedbea41cc6b449c455 P 44590e8af00e4d909dd1610f9f4a4bb3: Deleting consensus metadata
I20260929 18:32:26.277118 1630 ts_tablet_manager.cc:1918] T d36f265444044dedbea41cc6b449c455 P 215cca5e6a2d4d1aa326af4cd843fd48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:32:26.277326 1630 log.cc:1196] T d36f265444044dedbea41cc6b449c455 P 215cca5e6a2d4d1aa326af4cd843fd48: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790706718260899-29501-0/minicluster-data/ts-1/wal/wals/d36f265444044dedbea41cc6b449c455
I20260929 18:32:26.277726 1630 ts_tablet_manager.cc:1939] T d36f265444044dedbea41cc6b449c455 P 215cca5e6a2d4d1aa326af4cd843fd48: Deleting consensus metadata
I20260929 18:32:26.277927 1117 catalog_manager.cc:4841] TS 44590e8af00e4d909dd1610f9f4a4bb3 (127.28.207.67:45613): tablet d36f265444044dedbea41cc6b449c455 (table test-table [id=3a0524e233294ff4b1a4269c9890742a]) successfully deleted
I20260929 18:32:26.279546 1116 catalog_manager.cc:4841] TS 215cca5e6a2d4d1aa326af4cd843fd48 (127.28.207.66:38669): tablet d36f265444044dedbea41cc6b449c455 (table test-table [id=3a0524e233294ff4b1a4269c9890742a]) successfully deleted
I20260929 18:32:26.281975 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 1326
W20260929 18:32:26.281973 1117 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.207.65:34791 (error 108)
I20260929 18:32:26.290258 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 1459
I20260929 18:32:26.291340 1601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:26.290678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:34791 (local address 127.0.0.1:50256)
0929 18:32:26.291096 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:26.291196 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.65:34791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292}
W20260929 18:32:26.291780 1117 catalog_manager.cc:4584] TS 7656fb48bc0f4b68bf088a82abff9728 (127.28.207.65:34791): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d36f265444044dedbea41cc6b449c455: Network error: Client connection negotiation failed: client connection to 127.28.207.65:34791: connect: Connection refused (error 111)
I20260929 18:32:26.297166 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 1099
2026-09-29T18:32:26Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3935 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T18:32:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:32:26Z Disabled control of system clock
I20260929 18:32:26.331493 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:33349
--webserver_interface=127.28.207.126
--webserver_port=0
--builtin_ntp_servers=127.28.207.84:35349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:33349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 18:32:26.473320 1635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:26.473724 1635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:26.473783 1635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:26.478819 1635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:32:26.478942 1635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:26.478960 1635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:26.478978 1635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:32:26.478995 1635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:32:26.485296 1635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:35349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:33349
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:33349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:26.486581 1635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:26.488233 1635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:26.500180 1643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:26.500157 1641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:26.500180 1640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:26.500488 1635 server_base.cc:1034] running on GCE node
I20260929 18:32:26.501207 1635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:26.502457 1635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:26.503736 1635 hybrid_clock.cc:648] HybridClock initialized: now 1790706746503689 us; error 47 us; skew 500 ppm
I20260929 18:32:26.506657 1635 webserver.cc:460] Webserver started at http://127.28.207.126:46081/ using document root <none> and password file <none>
I20260929 18:32:26.507861 1635 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:26.507980 1635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:26.508293 1635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:26.511597 1635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "a24b37204bbe49cda64be3590a6e3008"
format_stamp: "Formatted at 2026-09-29 18:32:26 on dist-test-slave-qcpv"
server_key: "dfef25ac1c8d2bbdde28225d88e0e0af"
server_key_iv: "d4a08ac65447675dfddccb65a4f9c524"
server_key_version: "encryptionkey@0"
I20260929 18:32:26.512454 1635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "a24b37204bbe49cda64be3590a6e3008"
format_stamp: "Formatted at 2026-09-29 18:32:26 on dist-test-slave-qcpv"
server_key: "dfef25ac1c8d2bbdde28225d88e0e0af"
server_key_iv: "d4a08ac65447675dfddccb65a4f9c524"
server_key_version: "encryptionkey@0"
I20260929 18:32:26.517752 1635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 18:32:26.521551 1649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:26.523535 1635 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:32:26.523770 1635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "a24b37204bbe49cda64be3590a6e3008"
format_stamp: "Formatted at 2026-09-29 18:32:26 on dist-test-slave-qcpv"
server_key: "dfef25ac1c8d2bbdde28225d88e0e0af"
server_key_iv: "d4a08ac65447675dfddccb65a4f9c524"
server_key_version: "encryptionkey@0"
I20260929 18:32:26.523952 1635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:26.530083 1635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:26.531016 1635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:26.531265 1635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:26.542150 1635 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:33349
I20260929 18:32:26.542169 1701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:33349 every 8 connection(s)
I20260929 18:32:26.543740 1635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:32:26.547614 1702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:26.550074 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 1635
I20260929 18:32:26.550180 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:32:26.550540 29501 external_mini_cluster.cc:1428] Setting key f5c50f8636a70197f4020877a2caca85
I20260929 18:32:26.554546 1702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a24b37204bbe49cda64be3590a6e3008: Bootstrap starting.
I20260929 18:32:26.557440 1702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a24b37204bbe49cda64be3590a6e3008: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:26.558367 1702 log.cc:824] T 00000000000000000000000000000000 P a24b37204bbe49cda64be3590a6e3008: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:26.559043 1705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:26.552682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37884 (local address 127.28.207.126:33349)
0929 18:32:26.553633 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:26.553655 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:26.553707 (+ 52us) server_negotiation.cc:407] Connection header received
0929 18:32:26.554585 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:26.554608 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:26.555448 (+ 840us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:26.555860 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:26.557053 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:26.557059 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:26.557074 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:32:26.557091 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:26.557181 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:32:26.557201 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:26.557325 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:26.557500 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 18:32:26.557939 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":171,"threads_started":1}
I20260929 18:32:26.560974 1702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a24b37204bbe49cda64be3590a6e3008: No bootstrap required, opened a new log
I20260929 18:32:26.564826 1702 raft_consensus.cc:348] T 00000000000000000000000000000000 P a24b37204bbe49cda64be3590a6e3008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a24b37204bbe49cda64be3590a6e3008" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33349 } }
I20260929 18:32:26.565153 1702 raft_consensus.cc:374] T 00000000000000000000000000000000 P a24b37204bbe49cda64be3590a6e3008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:26.565248 1702 raft_consensus.cc:729] T 00000000000000000000000000000000 P a24b37204bbe49cda64be3590a6e3008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a24b37204bbe49cda64be3590a6e3008, State: Initialized, Role: FOLLOWER
I20260929 18:32:26.565693 1702 consensus_queue.cc:260] T 00000000000000000000000000000000 P a24b37204bbe49cda64be3590a6e3008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a24b37204bbe49cda64be3590a6e3008" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33349 } }
I20260929 18:32:26.565842 1702 raft_consensus.cc:388] T 00000000000000000000000000000000 P a24b37204bbe49cda64be3590a6e3008 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:32:26.565929 1702 raft_consensus.cc:482] T 00000000000000000000000000000000 P a24b37204bbe49cda64be3590a6e3008 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:32:26.566052 1702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a24b37204bbe49cda64be3590a6e3008 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:26.567160 1702 raft_consensus.cc:504] T 00000000000000000000000000000000 P a24b37204bbe49cda64be3590a6e3008 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a24b37204bbe49cda64be3590a6e3008" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33349 } }
I20260929 18:32:26.567510 1702 leader_election.cc:302] T 00000000000000000000000000000000 P a24b37204bbe49cda64be3590a6e3008 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a24b37204bbe49cda64be3590a6e3008; no voters:
I20260929 18:32:26.567859 1702 leader_election.cc:288] T 00000000000000000000000000000000 P a24b37204bbe49cda64be3590a6e3008 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:32:26.568110 1707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a24b37204bbe49cda64be3590a6e3008 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:32:26.568452 1707 raft_consensus.cc:686] T 00000000000000000000000000000000 P a24b37204bbe49cda64be3590a6e3008 [term 1 LEADER]: Becoming Leader. State: Replica: a24b37204bbe49cda64be3590a6e3008, State: Running, Role: LEADER
I20260929 18:32:26.568919 1707 consensus_queue.cc:237] T 00000000000000000000000000000000 P a24b37204bbe49cda64be3590a6e3008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a24b37204bbe49cda64be3590a6e3008" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33349 } }
I20260929 18:32:26.569375 1702 sys_catalog.cc:564] T 00000000000000000000000000000000 P a24b37204bbe49cda64be3590a6e3008 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:32:26.571281 1709 sys_catalog.cc:455] T 00000000000000000000000000000000 P a24b37204bbe49cda64be3590a6e3008 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a24b37204bbe49cda64be3590a6e3008. Latest consensus state: current_term: 1 leader_uuid: "a24b37204bbe49cda64be3590a6e3008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a24b37204bbe49cda64be3590a6e3008" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33349 } } }
I20260929 18:32:26.571313 1708 sys_catalog.cc:455] T 00000000000000000000000000000000 P a24b37204bbe49cda64be3590a6e3008 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a24b37204bbe49cda64be3590a6e3008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a24b37204bbe49cda64be3590a6e3008" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33349 } } }
I20260929 18:32:26.571422 1709 sys_catalog.cc:458] T 00000000000000000000000000000000 P a24b37204bbe49cda64be3590a6e3008 [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:26.571427 1708 sys_catalog.cc:458] T 00000000000000000000000000000000 P a24b37204bbe49cda64be3590a6e3008 [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:26.571810 1719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:32:26.574839 1719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:32:26.582813 1719 catalog_manager.cc:1347] Generated new cluster ID: ba1b2cb2a80e485d847685c17914da5a
I20260929 18:32:26.582966 1719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:32:26.595772 1719 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:32:26.596261 1719 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:32:26.620585 1719 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a24b37204bbe49cda64be3590a6e3008: Generated new TSK 0
I20260929 18:32:26.621870 1719 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:32:26.637264 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.65:0
--local_ip_for_outbound_sockets=127.28.207.65
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:33349
--builtin_ntp_servers=127.28.207.84:35349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 18:32:26.793797 1726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:26.794193 1726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:26.794268 1726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:26.798725 1726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:26.798852 1726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:26.799083 1726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.65
I20260929 18:32:26.804769 1726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:35349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:33349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.65
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:26.806511 1726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:26.808950 1726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:26.821722 1731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:26.821831 1726 server_base.cc:1034] running on GCE node
W20260929 18:32:26.821722 1734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:26.821746 1732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:26.822640 1726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:26.823454 1726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:26.824685 1726 hybrid_clock.cc:648] HybridClock initialized: now 1790706746824619 us; error 91 us; skew 500 ppm
I20260929 18:32:26.827184 1726 webserver.cc:460] Webserver started at http://127.28.207.65:35137/ using document root <none> and password file <none>
I20260929 18:32:26.827850 1726 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:26.827922 1726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:26.828138 1726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:26.830380 1726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-0/data/instance:
uuid: "a8594828b3e648edb2fcaaedc67c6c70"
format_stamp: "Formatted at 2026-09-29 18:32:26 on dist-test-slave-qcpv"
server_key: "ce2d578bab169c3836ac4f2742287f99"
server_key_iv: "624e3754a90f20f78f18c679dcfaffd0"
server_key_version: "encryptionkey@0"
I20260929 18:32:26.831028 1726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance:
uuid: "a8594828b3e648edb2fcaaedc67c6c70"
format_stamp: "Formatted at 2026-09-29 18:32:26 on dist-test-slave-qcpv"
server_key: "ce2d578bab169c3836ac4f2742287f99"
server_key_iv: "624e3754a90f20f78f18c679dcfaffd0"
server_key_version: "encryptionkey@0"
I20260929 18:32:26.835542 1726 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:32:26.839027 1740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:26.840915 1726 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:32:26.841071 1726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-0/wal
uuid: "a8594828b3e648edb2fcaaedc67c6c70"
format_stamp: "Formatted at 2026-09-29 18:32:26 on dist-test-slave-qcpv"
server_key: "ce2d578bab169c3836ac4f2742287f99"
server_key_iv: "624e3754a90f20f78f18c679dcfaffd0"
server_key_version: "encryptionkey@0"
I20260929 18:32:26.841259 1726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:26.847337 1726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:26.848083 1726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:26.848255 1726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:26.848929 1726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:32:26.849942 1726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:26.850001 1726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:26.850042 1726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:26.850057 1726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:26.861810 1726 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.65:33015
I20260929 18:32:26.861948 1853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.65:33015 every 8 connection(s)
I20260929 18:32:26.863041 1726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
I20260929 18:32:26.867731 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 1726
I20260929 18:32:26.867846 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance
I20260929 18:32:26.868155 29501 external_mini_cluster.cc:1428] Setting key e4077da1813cb6121c86650d680255b3
I20260929 18:32:26.871157 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.66:0
--local_ip_for_outbound_sockets=127.28.207.66
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:33349
--builtin_ntp_servers=127.28.207.84:35349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 18:32:26.871454 1705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:26.865586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:36869 (local address 127.28.207.126:33349)
0929 18:32:26.865807 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:26.865811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:26.867453 (+ 1642us) server_negotiation.cc:407] Connection header received
0929 18:32:26.869163 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:26.869167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:26.869213 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:26.869345 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:26.870905 (+ 1560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:26.870909 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:26.870912 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:32:26.870926 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:26.870961 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:32:26.870972 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:26.871086 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:26.871248 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 18:32:26.871307 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 18:32:26.872542 1856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:26.866124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:33349 (local address 127.28.207.65:36869)
0929 18:32:26.867202 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:26.867262 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:26.868924 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:26.869571 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:26.869580 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:26.869716 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:26.870518 (+ 802us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:32:26.870520 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:26.870567 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:32:26.870646 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:26.871069 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:26.871079 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:26.871086 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 18:32:26.871307 (+ 221us) client_negotiation.cc:240] Negotiation successful
0929 18:32:26.871592 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":159,"threads_started":1}
I20260929 18:32:26.874161 1854 heartbeater.cc:347] Connected to a master server at 127.28.207.126:33349
I20260929 18:32:26.874581 1854 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:26.875236 1854 heartbeater.cc:511] Master 127.28.207.126:33349 requested a full tablet report, sending...
I20260929 18:32:26.877483 1666 ts_manager.cc:194] Registered new tserver with Master: a8594828b3e648edb2fcaaedc67c6c70 (127.28.207.65:33015)
W20260929 18:32:27.019284 1857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:27.019635 1857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:27.019681 1857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:27.024113 1857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:27.024204 1857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:27.024268 1857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.66
I20260929 18:32:27.029574 1857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:35349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:33349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.66
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:27.031390 1857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:27.033581 1857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:27.047391 1863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:27.047348 1857 server_base.cc:1034] running on GCE node
W20260929 18:32:27.047284 1865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:27.047284 1862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:27.048404 1857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:27.049233 1857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:27.050467 1857 hybrid_clock.cc:648] HybridClock initialized: now 1790706747050450 us; error 35 us; skew 500 ppm
I20260929 18:32:27.053649 1857 webserver.cc:460] Webserver started at http://127.28.207.66:40105/ using document root <none> and password file <none>
I20260929 18:32:27.054351 1857 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:27.054453 1857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:27.054704 1857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:27.057147 1857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-1/data/instance:
uuid: "d2a598167f134fbc942e09d8a039411e"
format_stamp: "Formatted at 2026-09-29 18:32:27 on dist-test-slave-qcpv"
server_key: "e77a602e9f68b497a4b23d6f07ed5278"
server_key_iv: "9a3dcd1371e2df30eafd1725dc9711c4"
server_key_version: "encryptionkey@0"
I20260929 18:32:27.057849 1857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance:
uuid: "d2a598167f134fbc942e09d8a039411e"
format_stamp: "Formatted at 2026-09-29 18:32:27 on dist-test-slave-qcpv"
server_key: "e77a602e9f68b497a4b23d6f07ed5278"
server_key_iv: "9a3dcd1371e2df30eafd1725dc9711c4"
server_key_version: "encryptionkey@0"
I20260929 18:32:27.062434 1857 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 18:32:27.065691 1871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:27.067521 1857 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:32:27.067641 1857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-1/wal
uuid: "d2a598167f134fbc942e09d8a039411e"
format_stamp: "Formatted at 2026-09-29 18:32:27 on dist-test-slave-qcpv"
server_key: "e77a602e9f68b497a4b23d6f07ed5278"
server_key_iv: "9a3dcd1371e2df30eafd1725dc9711c4"
server_key_version: "encryptionkey@0"
I20260929 18:32:27.067750 1857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:27.073427 1857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:27.074088 1857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:27.074255 1857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:27.074784 1857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:32:27.075891 1857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:27.075947 1857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:27.075989 1857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:27.076004 1857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:27.086640 1857 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.66:39821
I20260929 18:32:27.086781 1984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.66:39821 every 8 connection(s)
I20260929 18:32:27.087805 1857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
I20260929 18:32:27.092840 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 1857
I20260929 18:32:27.093034 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance
I20260929 18:32:27.093374 29501 external_mini_cluster.cc:1428] Setting key cd504a04b5429ebd8e9817452dc77852
I20260929 18:32:27.096078 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.67:0
--local_ip_for_outbound_sockets=127.28.207.67
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:33349
--builtin_ntp_servers=127.28.207.84:35349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 18:32:27.096926 1705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:27.091273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:48269 (local address 127.28.207.126:33349)
0929 18:32:27.091489 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:27.091494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:27.092921 (+ 1427us) server_negotiation.cc:407] Connection header received
0929 18:32:27.094727 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:27.094731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:27.094782 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:27.094995 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:27.096402 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:27.096407 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:27.096410 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:32:27.096425 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:27.096455 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:32:27.096468 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:27.096535 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:27.096736 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 18:32:27.096785 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260929 18:32:27.097858 1987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:27.091814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:33349 (local address 127.28.207.66:48269)
0929 18:32:27.092710 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:27.092764 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:27.094446 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:27.095255 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:27.095268 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:27.095398 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:27.096050 (+ 652us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:32:27.096051 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:27.096094 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:32:27.096175 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:27.096597 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:27.096603 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:27.096607 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 18:32:27.096788 (+ 181us) client_negotiation.cc:240] Negotiation successful
0929 18:32:27.097050 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":152,"threads_started":1}
I20260929 18:32:27.099247 1985 heartbeater.cc:347] Connected to a master server at 127.28.207.126:33349
I20260929 18:32:27.099627 1985 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:27.100276 1985 heartbeater.cc:511] Master 127.28.207.126:33349 requested a full tablet report, sending...
I20260929 18:32:27.101763 1666 ts_manager.cc:194] Registered new tserver with Master: d2a598167f134fbc942e09d8a039411e (127.28.207.66:39821)
W20260929 18:32:27.243636 1988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:27.244051 1988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:27.244107 1988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:27.249897 1988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:27.249993 1988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:27.250097 1988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.67
I20260929 18:32:27.257269 1988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:35349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:33349
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.67
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:27.258987 1988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:27.260816 1988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:27.273948 1996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:27.274071 1994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:27.273948 1993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:27.274356 1988 server_base.cc:1034] running on GCE node
I20260929 18:32:27.275187 1988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:27.276054 1988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:27.277269 1988 hybrid_clock.cc:648] HybridClock initialized: now 1790706747277250 us; error 37 us; skew 500 ppm
I20260929 18:32:27.280117 1988 webserver.cc:460] Webserver started at http://127.28.207.67:34731/ using document root <none> and password file <none>
I20260929 18:32:27.280841 1988 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:27.280915 1988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:27.281128 1988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:27.283843 1988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-2/data/instance:
uuid: "75b5d50177174021a2e543b56ee7e8fe"
format_stamp: "Formatted at 2026-09-29 18:32:27 on dist-test-slave-qcpv"
server_key: "07b164dcb9ea1a2417624aecb007ef17"
server_key_iv: "d5544d8804d5eadd2136e4422e7d3266"
server_key_version: "encryptionkey@0"
I20260929 18:32:27.284612 1988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance:
uuid: "75b5d50177174021a2e543b56ee7e8fe"
format_stamp: "Formatted at 2026-09-29 18:32:27 on dist-test-slave-qcpv"
server_key: "07b164dcb9ea1a2417624aecb007ef17"
server_key_iv: "d5544d8804d5eadd2136e4422e7d3266"
server_key_version: "encryptionkey@0"
I20260929 18:32:27.292785 1988 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260929 18:32:27.296674 2002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:27.298425 1988 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 18:32:27.298615 1988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-2/wal
uuid: "75b5d50177174021a2e543b56ee7e8fe"
format_stamp: "Formatted at 2026-09-29 18:32:27 on dist-test-slave-qcpv"
server_key: "07b164dcb9ea1a2417624aecb007ef17"
server_key_iv: "d5544d8804d5eadd2136e4422e7d3266"
server_key_version: "encryptionkey@0"
I20260929 18:32:27.298774 1988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:27.305253 1988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:27.306088 1988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:27.306321 1988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:27.307145 1988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:32:27.308320 1988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:27.308396 1988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:27.308471 1988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:27.308521 1988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:27.321619 1988 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.67:43717
I20260929 18:32:27.321641 2115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.67:43717 every 8 connection(s)
I20260929 18:32:27.322824 1988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
I20260929 18:32:27.326164 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 1988
I20260929 18:32:27.326313 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance
I20260929 18:32:27.326612 29501 external_mini_cluster.cc:1428] Setting key 2d9b4ef693c0300e3d4860c69a2dc53d
I20260929 18:32:27.331348 1705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:27.325219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:35791 (local address 127.28.207.126:33349)
0929 18:32:27.325425 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:27.325429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:27.327191 (+ 1762us) server_negotiation.cc:407] Connection header received
0929 18:32:27.329328 (+ 2137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:27.329334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:27.329391 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:27.329475 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:27.330818 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:27.330821 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:27.330824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:32:27.330836 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:27.330881 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:32:27.330889 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:27.330993 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:27.331184 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 18:32:27.331225 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260929 18:32:27.332688 2118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:27.325769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:33349 (local address 127.28.207.67:35791)
0929 18:32:27.326941 (+ 1172us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:27.327020 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:27.329011 (+ 1991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:27.329703 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:27.329712 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:27.329823 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:27.330475 (+ 652us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:32:27.330476 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:27.330536 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:32:27.330623 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:27.330997 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:27.331023 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:27.331029 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:32:27.331216 (+ 187us) client_negotiation.cc:240] Negotiation successful
0929 18:32:27.331527 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":192,"threads_started":1}
I20260929 18:32:27.334556 2116 heartbeater.cc:347] Connected to a master server at 127.28.207.126:33349
I20260929 18:32:27.335017 2116 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:27.335597 2116 heartbeater.cc:511] Master 127.28.207.126:33349 requested a full tablet report, sending...
I20260929 18:32:27.336931 1666 ts_manager.cc:194] Registered new tserver with Master: 75b5d50177174021a2e543b56ee7e8fe (127.28.207.67:43717)
I20260929 18:32:27.341929 29501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:32:27.347362 1705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:27.344793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37900 (local address 127.28.207.126:33349)
0929 18:32:27.344930 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:27.344934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:27.345373 (+ 439us) server_negotiation.cc:407] Connection header received
0929 18:32:27.345551 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:27.345554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:27.345597 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:27.345715 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:27.346788 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:27.346792 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:27.346794 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:32:27.346806 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:27.346831 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:32:27.346839 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:27.347016 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:27.347199 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 18:32:27.347241 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:32:27.351795 1666 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:32:27.356132 1666 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:32:27.373158 2127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:27.368691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:43717 (local address 127.0.0.1:39912)
0929 18:32:27.369642 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:27.369658 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:27.369774 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:27.370438 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:27.370445 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:27.370528 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:27.372195 (+ 1667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:32:27.372197 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:27.372220 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:32:27.372306 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:27.372708 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:27.372718 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:27.372725 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 18:32:27.372843 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 18:32:27.372929 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"thread_start_us":122,"threads_started":1}
I20260929 18:32:27.373749 2131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:27.369222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:33015 (local address 127.0.0.1:51892)
0929 18:32:27.369783 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:27.369799 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:27.370312 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:27.372183 (+ 1871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:27.372188 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:27.372208 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:27.372761 (+ 553us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:32:27.372763 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:27.372774 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:32:27.372830 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:27.373204 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:27.373208 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:27.373213 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:32:27.373588 (+ 375us) client_negotiation.cc:240] Negotiation successful
0929 18:32:27.373639 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":104,"threads_started":1}
I20260929 18:32:27.373898 2132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:27.369654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51892 (local address 127.28.207.65:33015)
0929 18:32:27.370048 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:27.370055 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:27.370183 (+ 128us) server_negotiation.cc:407] Connection header received
0929 18:32:27.371028 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:27.371037 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:27.371230 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:27.371530 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:27.372992 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:27.372999 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:27.373013 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:32:27.373026 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:27.373072 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:32:27.373107 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:27.373237 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:27.373631 (+ 394us) server_negotiation.cc:299] Negotiation successful
0929 18:32:27.373720 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":104,"threads_started":1}
I20260929 18:32:27.373957 2130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:27.368867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39912 (local address 127.28.207.67:43717)
0929 18:32:27.369526 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:27.369564 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:27.369901 (+ 337us) server_negotiation.cc:407] Connection header received
0929 18:32:27.370020 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:27.370026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:27.370274 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:27.370442 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:27.372436 (+ 1994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:27.372448 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:27.372463 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:32:27.372479 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:27.372527 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:32:27.372557 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:27.372662 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:27.373616 (+ 954us) server_negotiation.cc:299] Negotiation successful
0929 18:32:27.373731 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":114,"threads_started":1}
I20260929 18:32:27.374037 2128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:27.368401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49888 (local address 127.28.207.66:39821)
0929 18:32:27.369425 (+ 1024us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:27.369440 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:27.369624 (+ 184us) server_negotiation.cc:407] Connection header received
0929 18:32:27.370314 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:27.370323 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:27.371100 (+ 777us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:27.371251 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:27.373339 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:27.373345 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:27.373385 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:32:27.373401 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:27.373447 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:32:27.373479 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:27.373561 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:27.373807 (+ 246us) server_negotiation.cc:299] Negotiation successful
0929 18:32:27.373875 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":889,"thread_start_us":205,"threads_started":1}
I20260929 18:32:27.374390 2129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:27.368216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:39821 (local address 127.0.0.1:49888)
0929 18:32:27.369300 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:27.369381 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:27.369733 (+ 352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:27.372429 (+ 2696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:27.372434 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:27.372454 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:27.373151 (+ 697us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:32:27.373154 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:27.373167 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:32:27.373195 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:27.373697 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:27.373700 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:27.373702 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 18:32:27.374196 (+ 494us) client_negotiation.cc:240] Negotiation successful
0929 18:32:27.374243 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"thread_start_us":99,"threads_started":1}
I20260929 18:32:27.377949 1788 tablet_service.cc:1467] Processing CreateTablet for tablet 5999c845ecdf42b199b7bd4328ed3891 (DEFAULT_TABLE table=test-table [id=5e95965bf0834a6996c8fc2e9351e720]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:27.377933 2050 tablet_service.cc:1467] Processing CreateTablet for tablet 5999c845ecdf42b199b7bd4328ed3891 (DEFAULT_TABLE table=test-table [id=5e95965bf0834a6996c8fc2e9351e720]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:27.379419 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5999c845ecdf42b199b7bd4328ed3891. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:27.379419 2050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5999c845ecdf42b199b7bd4328ed3891. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:27.380614 1919 tablet_service.cc:1467] Processing CreateTablet for tablet 5999c845ecdf42b199b7bd4328ed3891 (DEFAULT_TABLE table=test-table [id=5e95965bf0834a6996c8fc2e9351e720]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:27.381664 1919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5999c845ecdf42b199b7bd4328ed3891. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:27.388605 2133 tablet_bootstrap.cc:492] T 5999c845ecdf42b199b7bd4328ed3891 P a8594828b3e648edb2fcaaedc67c6c70: Bootstrap starting.
I20260929 18:32:27.389400 2134 tablet_bootstrap.cc:492] T 5999c845ecdf42b199b7bd4328ed3891 P 75b5d50177174021a2e543b56ee7e8fe: Bootstrap starting.
I20260929 18:32:27.389605 2135 tablet_bootstrap.cc:492] T 5999c845ecdf42b199b7bd4328ed3891 P d2a598167f134fbc942e09d8a039411e: Bootstrap starting.
I20260929 18:32:27.392031 2135 tablet_bootstrap.cc:654] T 5999c845ecdf42b199b7bd4328ed3891 P d2a598167f134fbc942e09d8a039411e: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:27.392338 2133 tablet_bootstrap.cc:654] T 5999c845ecdf42b199b7bd4328ed3891 P a8594828b3e648edb2fcaaedc67c6c70: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:27.392338 2134 tablet_bootstrap.cc:654] T 5999c845ecdf42b199b7bd4328ed3891 P 75b5d50177174021a2e543b56ee7e8fe: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:27.392980 2135 log.cc:824] T 5999c845ecdf42b199b7bd4328ed3891 P d2a598167f134fbc942e09d8a039411e: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:27.393374 2133 log.cc:824] T 5999c845ecdf42b199b7bd4328ed3891 P a8594828b3e648edb2fcaaedc67c6c70: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:27.393374 2134 log.cc:824] T 5999c845ecdf42b199b7bd4328ed3891 P 75b5d50177174021a2e543b56ee7e8fe: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:27.395949 2133 tablet_bootstrap.cc:492] T 5999c845ecdf42b199b7bd4328ed3891 P a8594828b3e648edb2fcaaedc67c6c70: No bootstrap required, opened a new log
I20260929 18:32:27.395969 2135 tablet_bootstrap.cc:492] T 5999c845ecdf42b199b7bd4328ed3891 P d2a598167f134fbc942e09d8a039411e: No bootstrap required, opened a new log
I20260929 18:32:27.396078 2134 tablet_bootstrap.cc:492] T 5999c845ecdf42b199b7bd4328ed3891 P 75b5d50177174021a2e543b56ee7e8fe: No bootstrap required, opened a new log
I20260929 18:32:27.396216 2133 ts_tablet_manager.cc:1397] T 5999c845ecdf42b199b7bd4328ed3891 P a8594828b3e648edb2fcaaedc67c6c70: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.004s
I20260929 18:32:27.396238 2134 ts_tablet_manager.cc:1397] T 5999c845ecdf42b199b7bd4328ed3891 P 75b5d50177174021a2e543b56ee7e8fe: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 18:32:27.396283 2135 ts_tablet_manager.cc:1397] T 5999c845ecdf42b199b7bd4328ed3891 P d2a598167f134fbc942e09d8a039411e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 18:32:27.401070 2135 raft_consensus.cc:348] T 5999c845ecdf42b199b7bd4328ed3891 P d2a598167f134fbc942e09d8a039411e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2a598167f134fbc942e09d8a039411e" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 39821 } } peers { permanent_uuid: "a8594828b3e648edb2fcaaedc67c6c70" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 33015 } } peers { permanent_uuid: "75b5d50177174021a2e543b56ee7e8fe" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 43717 } }
I20260929 18:32:27.401428 2135 raft_consensus.cc:374] T 5999c845ecdf42b199b7bd4328ed3891 P d2a598167f134fbc942e09d8a039411e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:27.401487 2135 raft_consensus.cc:729] T 5999c845ecdf42b199b7bd4328ed3891 P d2a598167f134fbc942e09d8a039411e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2a598167f134fbc942e09d8a039411e, State: Initialized, Role: FOLLOWER
I20260929 18:32:27.401465 2133 raft_consensus.cc:348] T 5999c845ecdf42b199b7bd4328ed3891 P a8594828b3e648edb2fcaaedc67c6c70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2a598167f134fbc942e09d8a039411e" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 39821 } } peers { permanent_uuid: "a8594828b3e648edb2fcaaedc67c6c70" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 33015 } } peers { permanent_uuid: "75b5d50177174021a2e543b56ee7e8fe" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 43717 } }
I20260929 18:32:27.401465 2134 raft_consensus.cc:348] T 5999c845ecdf42b199b7bd4328ed3891 P 75b5d50177174021a2e543b56ee7e8fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2a598167f134fbc942e09d8a039411e" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 39821 } } peers { permanent_uuid: "a8594828b3e648edb2fcaaedc67c6c70" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 33015 } } peers { permanent_uuid: "75b5d50177174021a2e543b56ee7e8fe" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 43717 } }
I20260929 18:32:27.401808 2133 raft_consensus.cc:374] T 5999c845ecdf42b199b7bd4328ed3891 P a8594828b3e648edb2fcaaedc67c6c70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:27.401808 2134 raft_consensus.cc:374] T 5999c845ecdf42b199b7bd4328ed3891 P 75b5d50177174021a2e543b56ee7e8fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:27.401871 2133 raft_consensus.cc:729] T 5999c845ecdf42b199b7bd4328ed3891 P a8594828b3e648edb2fcaaedc67c6c70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8594828b3e648edb2fcaaedc67c6c70, State: Initialized, Role: FOLLOWER
I20260929 18:32:27.401880 2134 raft_consensus.cc:729] T 5999c845ecdf42b199b7bd4328ed3891 P 75b5d50177174021a2e543b56ee7e8fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75b5d50177174021a2e543b56ee7e8fe, State: Initialized, Role: FOLLOWER
I20260929 18:32:27.402091 2135 consensus_queue.cc:260] T 5999c845ecdf42b199b7bd4328ed3891 P d2a598167f134fbc942e09d8a039411e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2a598167f134fbc942e09d8a039411e" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 39821 } } peers { permanent_uuid: "a8594828b3e648edb2fcaaedc67c6c70" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 33015 } } peers { permanent_uuid: "75b5d50177174021a2e543b56ee7e8fe" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 43717 } }
I20260929 18:32:27.402487 2133 consensus_queue.cc:260] T 5999c845ecdf42b199b7bd4328ed3891 P a8594828b3e648edb2fcaaedc67c6c70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2a598167f134fbc942e09d8a039411e" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 39821 } } peers { permanent_uuid: "a8594828b3e648edb2fcaaedc67c6c70" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 33015 } } peers { permanent_uuid: "75b5d50177174021a2e543b56ee7e8fe" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 43717 } }
I20260929 18:32:27.402488 2134 consensus_queue.cc:260] T 5999c845ecdf42b199b7bd4328ed3891 P 75b5d50177174021a2e543b56ee7e8fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2a598167f134fbc942e09d8a039411e" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 39821 } } peers { permanent_uuid: "a8594828b3e648edb2fcaaedc67c6c70" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 33015 } } peers { permanent_uuid: "75b5d50177174021a2e543b56ee7e8fe" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 43717 } }
I20260929 18:32:27.403184 1985 heartbeater.cc:503] Master 127.28.207.126:33349 was elected leader, sending a full tablet report...
I20260929 18:32:27.403734 2134 ts_tablet_manager.cc:1428] T 5999c845ecdf42b199b7bd4328ed3891 P 75b5d50177174021a2e543b56ee7e8fe: Time spent starting tablet: real 0.007s user 0.005s sys 0.003s
I20260929 18:32:27.403734 2133 ts_tablet_manager.cc:1428] T 5999c845ecdf42b199b7bd4328ed3891 P a8594828b3e648edb2fcaaedc67c6c70: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260929 18:32:27.404194 2116 heartbeater.cc:503] Master 127.28.207.126:33349 was elected leader, sending a full tablet report...
I20260929 18:32:27.404544 1854 heartbeater.cc:503] Master 127.28.207.126:33349 was elected leader, sending a full tablet report...
I20260929 18:32:27.406118 2135 ts_tablet_manager.cc:1428] T 5999c845ecdf42b199b7bd4328ed3891 P d2a598167f134fbc942e09d8a039411e: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20260929 18:32:27.462615 2140 raft_consensus.cc:482] T 5999c845ecdf42b199b7bd4328ed3891 P a8594828b3e648edb2fcaaedc67c6c70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:32:27.462980 2140 raft_consensus.cc:504] T 5999c845ecdf42b199b7bd4328ed3891 P a8594828b3e648edb2fcaaedc67c6c70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2a598167f134fbc942e09d8a039411e" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 39821 } } peers { permanent_uuid: "a8594828b3e648edb2fcaaedc67c6c70" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 33015 } } peers { permanent_uuid: "75b5d50177174021a2e543b56ee7e8fe" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 43717 } }
I20260929 18:32:27.464486 2140 leader_election.cc:288] T 5999c845ecdf42b199b7bd4328ed3891 P a8594828b3e648edb2fcaaedc67c6c70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2a598167f134fbc942e09d8a039411e (127.28.207.66:39821), 75b5d50177174021a2e543b56ee7e8fe (127.28.207.67:43717)
I20260929 18:32:27.467306 2143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:27.464866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:39821 (local address 127.28.207.65:45987)
0929 18:32:27.465281 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:27.465298 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:27.465433 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:27.465719 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:27.465722 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:27.465736 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:27.466294 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:32:27.466296 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:27.466307 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:32:27.466336 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:27.466932 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:27.466936 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:27.466939 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:32:27.467037 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 18:32:27.467080 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":154,"threads_started":1}
I20260929 18:32:27.467299 2128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:27.464989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:45987 (local address 127.28.207.66:39821)
0929 18:32:27.465347 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:27.465354 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:27.465372 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:32:27.465545 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:27.465548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:27.465593 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:27.465699 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:27.466579 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:27.466585 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:27.466589 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:32:27.466608 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:27.466639 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:32:27.466760 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:27.466947 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:27.467035 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 18:32:27.467077 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244}
I20260929 18:32:27.467439 2130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:27.465018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:46221 (local address 127.28.207.67:43717)
0929 18:32:27.465289 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:27.465297 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:27.465749 (+ 452us) server_negotiation.cc:407] Connection header received
0929 18:32:27.465960 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:27.465965 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:27.466017 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:27.466116 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:27.466968 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:27.466975 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:27.466982 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:32:27.467001 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:27.467032 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:32:27.467044 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:27.467134 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:27.467258 (+ 124us) server_negotiation.cc:299] Negotiation successful
0929 18:32:27.467304 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260929 18:32:27.467774 2142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:27.464878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:43717 (local address 127.28.207.65:46221)
0929 18:32:27.465471 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:27.465483 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:27.465829 (+ 346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:27.466136 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:27.466139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:27.466157 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:27.466635 (+ 478us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:32:27.466637 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:27.466661 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:32:27.466806 (+ 145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:27.467135 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:27.467138 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:27.467141 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 18:32:27.467591 (+ 450us) client_negotiation.cc:240] Negotiation successful
0929 18:32:27.467632 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":160,"threads_started":1}
I20260929 18:32:27.468211 1939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5999c845ecdf42b199b7bd4328ed3891" candidate_uuid: "a8594828b3e648edb2fcaaedc67c6c70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2a598167f134fbc942e09d8a039411e" is_pre_election: true
I20260929 18:32:27.468590 1939 raft_consensus.cc:2454] T 5999c845ecdf42b199b7bd4328ed3891 P d2a598167f134fbc942e09d8a039411e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8594828b3e648edb2fcaaedc67c6c70 in term 0.
I20260929 18:32:27.468567 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5999c845ecdf42b199b7bd4328ed3891" candidate_uuid: "a8594828b3e648edb2fcaaedc67c6c70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b5d50177174021a2e543b56ee7e8fe" is_pre_election: true
I20260929 18:32:27.468953 2070 raft_consensus.cc:2454] T 5999c845ecdf42b199b7bd4328ed3891 P 75b5d50177174021a2e543b56ee7e8fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8594828b3e648edb2fcaaedc67c6c70 in term 0.
I20260929 18:32:27.469141 1744 leader_election.cc:302] T 5999c845ecdf42b199b7bd4328ed3891 P a8594828b3e648edb2fcaaedc67c6c70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8594828b3e648edb2fcaaedc67c6c70, d2a598167f134fbc942e09d8a039411e; no voters:
I20260929 18:32:27.469565 2140 raft_consensus.cc:2781] T 5999c845ecdf42b199b7bd4328ed3891 P a8594828b3e648edb2fcaaedc67c6c70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:32:27.469806 2140 raft_consensus.cc:482] T 5999c845ecdf42b199b7bd4328ed3891 P a8594828b3e648edb2fcaaedc67c6c70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:32:27.469892 2140 raft_consensus.cc:3037] T 5999c845ecdf42b199b7bd4328ed3891 P a8594828b3e648edb2fcaaedc67c6c70 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:27.471814 2140 raft_consensus.cc:504] T 5999c845ecdf42b199b7bd4328ed3891 P a8594828b3e648edb2fcaaedc67c6c70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2a598167f134fbc942e09d8a039411e" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 39821 } } peers { permanent_uuid: "a8594828b3e648edb2fcaaedc67c6c70" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 33015 } } peers { permanent_uuid: "75b5d50177174021a2e543b56ee7e8fe" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 43717 } }
I20260929 18:32:27.472345 2140 leader_election.cc:288] T 5999c845ecdf42b199b7bd4328ed3891 P a8594828b3e648edb2fcaaedc67c6c70 [CANDIDATE]: Term 1 election: Requested vote from peers d2a598167f134fbc942e09d8a039411e (127.28.207.66:39821), 75b5d50177174021a2e543b56ee7e8fe (127.28.207.67:43717)
I20260929 18:32:27.472926 2070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5999c845ecdf42b199b7bd4328ed3891" candidate_uuid: "a8594828b3e648edb2fcaaedc67c6c70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b5d50177174021a2e543b56ee7e8fe"
I20260929 18:32:27.472927 1939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5999c845ecdf42b199b7bd4328ed3891" candidate_uuid: "a8594828b3e648edb2fcaaedc67c6c70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2a598167f134fbc942e09d8a039411e"
I20260929 18:32:27.473107 1939 raft_consensus.cc:3037] T 5999c845ecdf42b199b7bd4328ed3891 P d2a598167f134fbc942e09d8a039411e [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:27.473107 2070 raft_consensus.cc:3037] T 5999c845ecdf42b199b7bd4328ed3891 P 75b5d50177174021a2e543b56ee7e8fe [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:27.475337 2070 raft_consensus.cc:2454] T 5999c845ecdf42b199b7bd4328ed3891 P 75b5d50177174021a2e543b56ee7e8fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8594828b3e648edb2fcaaedc67c6c70 in term 1.
I20260929 18:32:27.475337 1939 raft_consensus.cc:2454] T 5999c845ecdf42b199b7bd4328ed3891 P d2a598167f134fbc942e09d8a039411e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8594828b3e648edb2fcaaedc67c6c70 in term 1.
I20260929 18:32:27.475831 1744 leader_election.cc:302] T 5999c845ecdf42b199b7bd4328ed3891 P a8594828b3e648edb2fcaaedc67c6c70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 75b5d50177174021a2e543b56ee7e8fe, a8594828b3e648edb2fcaaedc67c6c70, d2a598167f134fbc942e09d8a039411e; no voters:
I20260929 18:32:27.476158 2140 raft_consensus.cc:2781] T 5999c845ecdf42b199b7bd4328ed3891 P a8594828b3e648edb2fcaaedc67c6c70 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:32:27.476449 2140 raft_consensus.cc:686] T 5999c845ecdf42b199b7bd4328ed3891 P a8594828b3e648edb2fcaaedc67c6c70 [term 1 LEADER]: Becoming Leader. State: Replica: a8594828b3e648edb2fcaaedc67c6c70, State: Running, Role: LEADER
I20260929 18:32:27.476843 2140 consensus_queue.cc:237] T 5999c845ecdf42b199b7bd4328ed3891 P a8594828b3e648edb2fcaaedc67c6c70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2a598167f134fbc942e09d8a039411e" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 39821 } } peers { permanent_uuid: "a8594828b3e648edb2fcaaedc67c6c70" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 33015 } } peers { permanent_uuid: "75b5d50177174021a2e543b56ee7e8fe" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 43717 } }
I20260929 18:32:27.481321 1666 catalog_manager.cc:5495] T 5999c845ecdf42b199b7bd4328ed3891 P a8594828b3e648edb2fcaaedc67c6c70 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8594828b3e648edb2fcaaedc67c6c70 (127.28.207.65). New cstate: current_term: 1 leader_uuid: "a8594828b3e648edb2fcaaedc67c6c70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2a598167f134fbc942e09d8a039411e" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 39821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8594828b3e648edb2fcaaedc67c6c70" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 33015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75b5d50177174021a2e543b56ee7e8fe" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 43717 } health_report { overall_health: UNKNOWN } } }
I20260929 18:32:27.500859 2132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:27.499079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51896 (local address 127.28.207.65:33015)
0929 18:32:27.499286 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:27.499292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:27.499307 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:32:27.499490 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:27.499495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:27.499582 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:27.499678 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:27.500441 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:27.500444 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:27.500447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:32:27.500457 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:27.500476 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:32:27.500485 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:27.500592 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:27.500668 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 18:32:27.500706 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260929 18:32:27.508327 1939 raft_consensus.cc:1261] T 5999c845ecdf42b199b7bd4328ed3891 P d2a598167f134fbc942e09d8a039411e [term 1 FOLLOWER]: Refusing update from remote peer a8594828b3e648edb2fcaaedc67c6c70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:32:27.508370 2070 raft_consensus.cc:1261] T 5999c845ecdf42b199b7bd4328ed3891 P 75b5d50177174021a2e543b56ee7e8fe [term 1 FOLLOWER]: Refusing update from remote peer a8594828b3e648edb2fcaaedc67c6c70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:32:27.509362 2140 consensus_queue.cc:1035] T 5999c845ecdf42b199b7bd4328ed3891 P a8594828b3e648edb2fcaaedc67c6c70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2a598167f134fbc942e09d8a039411e" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 39821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:32:27.509788 2144 consensus_queue.cc:1035] T 5999c845ecdf42b199b7bd4328ed3891 P a8594828b3e648edb2fcaaedc67c6c70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b5d50177174021a2e543b56ee7e8fe" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 43717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:32:27.520907 2148 mvcc.cc:205] Tried to move back new op lower bound from 7334734837783994368 to 7334734837668581376. Current Snapshot: MvccSnapshot[applied={T|T < 7334734837783994368}]
I20260929 18:32:27.534684 1666 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:32:27.535043 1666 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:32:27.538232 1666 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a24b37204bbe49cda64be3590a6e3008: Sending DeleteTablet for 3 replicas of tablet 5999c845ecdf42b199b7bd4328ed3891
I20260929 18:32:27.539454 2050 tablet_service.cc:1514] Processing DeleteTablet for tablet 5999c845ecdf42b199b7bd4328ed3891 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:27 UTC) from {username='slave'} at 127.0.0.1:39912
I20260929 18:32:27.539459 1919 tablet_service.cc:1514] Processing DeleteTablet for tablet 5999c845ecdf42b199b7bd4328ed3891 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:27 UTC) from {username='slave'} at 127.0.0.1:49888
I20260929 18:32:27.539674 1788 tablet_service.cc:1514] Processing DeleteTablet for tablet 5999c845ecdf42b199b7bd4328ed3891 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:27 UTC) from {username='slave'} at 127.0.0.1:51892
I20260929 18:32:27.540288 2160 tablet_replica.cc:330] T 5999c845ecdf42b199b7bd4328ed3891 P a8594828b3e648edb2fcaaedc67c6c70: stopping tablet replica
I20260929 18:32:27.540573 2158 tablet_replica.cc:330] T 5999c845ecdf42b199b7bd4328ed3891 P 75b5d50177174021a2e543b56ee7e8fe: stopping tablet replica
I20260929 18:32:27.540658 2160 raft_consensus.cc:2229] T 5999c845ecdf42b199b7bd4328ed3891 P a8594828b3e648edb2fcaaedc67c6c70 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:32:27.540763 2159 tablet_replica.cc:330] T 5999c845ecdf42b199b7bd4328ed3891 P d2a598167f134fbc942e09d8a039411e: stopping tablet replica
I20260929 18:32:27.541115 2160 raft_consensus.cc:2258] T 5999c845ecdf42b199b7bd4328ed3891 P a8594828b3e648edb2fcaaedc67c6c70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:27.541491 2159 raft_consensus.cc:2229] T 5999c845ecdf42b199b7bd4328ed3891 P d2a598167f134fbc942e09d8a039411e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:32:27.541555 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 1726
I20260929 18:32:27.541944 2159 raft_consensus.cc:2258] T 5999c845ecdf42b199b7bd4328ed3891 P d2a598167f134fbc942e09d8a039411e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:27.542250 2158 raft_consensus.cc:2229] T 5999c845ecdf42b199b7bd4328ed3891 P 75b5d50177174021a2e543b56ee7e8fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:32:27.542577 2158 raft_consensus.cc:2258] T 5999c845ecdf42b199b7bd4328ed3891 P 75b5d50177174021a2e543b56ee7e8fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:27.544862 2158 ts_tablet_manager.cc:1905] T 5999c845ecdf42b199b7bd4328ed3891 P 75b5d50177174021a2e543b56ee7e8fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:32:27.546532 2159 ts_tablet_manager.cc:1905] T 5999c845ecdf42b199b7bd4328ed3891 P d2a598167f134fbc942e09d8a039411e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:32:27.549229 2158 ts_tablet_manager.cc:1918] T 5999c845ecdf42b199b7bd4328ed3891 P 75b5d50177174021a2e543b56ee7e8fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:32:27.549372 2158 log.cc:1196] T 5999c845ecdf42b199b7bd4328ed3891 P 75b5d50177174021a2e543b56ee7e8fe: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-2/wal/wals/5999c845ecdf42b199b7bd4328ed3891
I20260929 18:32:27.549760 2158 ts_tablet_manager.cc:1939] T 5999c845ecdf42b199b7bd4328ed3891 P 75b5d50177174021a2e543b56ee7e8fe: Deleting consensus metadata
I20260929 18:32:27.549777 2159 ts_tablet_manager.cc:1918] T 5999c845ecdf42b199b7bd4328ed3891 P d2a598167f134fbc942e09d8a039411e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:32:27.550057 2159 log.cc:1196] T 5999c845ecdf42b199b7bd4328ed3891 P d2a598167f134fbc942e09d8a039411e: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790706718260899-29501-0/minicluster-data/ts-1/wal/wals/5999c845ecdf42b199b7bd4328ed3891
I20260929 18:32:27.550426 2159 ts_tablet_manager.cc:1939] T 5999c845ecdf42b199b7bd4328ed3891 P d2a598167f134fbc942e09d8a039411e: Deleting consensus metadata
I20260929 18:32:27.551581 1650 catalog_manager.cc:4841] TS 75b5d50177174021a2e543b56ee7e8fe (127.28.207.67:43717): tablet 5999c845ecdf42b199b7bd4328ed3891 (table test-table [id=5e95965bf0834a6996c8fc2e9351e720]) successfully deleted
I20260929 18:32:27.551898 1653 catalog_manager.cc:4841] TS d2a598167f134fbc942e09d8a039411e (127.28.207.66:39821): tablet 5999c845ecdf42b199b7bd4328ed3891 (table test-table [id=5e95965bf0834a6996c8fc2e9351e720]) successfully deleted
W20260929 18:32:27.552922 1653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.207.65:33015 (error 108)
I20260929 18:32:27.553916 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 1857
I20260929 18:32:27.554229 2131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:27.553870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:33015 (local address 127.0.0.1:51898)
0929 18:32:27.554030 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:27.554114 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.65:33015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 18:32:27.555848 1653 catalog_manager.cc:4584] TS a8594828b3e648edb2fcaaedc67c6c70 (127.28.207.65:33015): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5999c845ecdf42b199b7bd4328ed3891: Network error: Client connection negotiation failed: client connection to 127.28.207.65:33015: connect: Connection refused (error 111)
I20260929 18:32:27.561861 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 1988
I20260929 18:32:27.571512 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 1635
2026-09-29T18:32:27Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1276 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 18:32:27.597492 29501 mini_oidc.cc:152] Starting JWKS server
I20260929 18:32:27.599645 29501 webserver.cc:460] Webserver started at https://127.0.0.1:37899/ using document root <none> and password file <none>
I20260929 18:32:27.599818 29501 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 18:32:27.600071 29501 webserver.cc:460] Webserver started at http://127.0.0.1:40389/ using document root <none> and password file <none>
2026-09-29T18:32:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:32:27Z Disabled control of system clock
I20260929 18:32:27.612829 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:35415
--webserver_interface=127.28.207.126
--webserver_port=0
--builtin_ntp_servers=127.28.207.84:40199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:35415
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37899/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706718260899-29501-0/testchainca.cert with env {}
W20260929 18:32:27.732936 2167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:27.733260 2167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:27.733302 2167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:27.737483 2167 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:32:27.737565 2167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:27.737582 2167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:27.737599 2167 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:32:27.737617 2167 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 18:32:27.737633 2167 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 18:32:27.737648 2167 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37899/jwks.json
I20260929 18:32:27.743448 2167 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:40199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:35415
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:35415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37899/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706718260899-29501-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:27.744613 2167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:27.745647 2167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:27.756004 2173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:27.756115 2167 server_base.cc:1034] running on GCE node
W20260929 18:32:27.756047 2172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:27.755990 2175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:27.757344 2167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:27.758831 2167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:27.760131 2167 hybrid_clock.cc:648] HybridClock initialized: now 1790706747760111 us; error 46 us; skew 500 ppm
I20260929 18:32:27.762725 2167 webserver.cc:460] Webserver started at http://127.28.207.126:32889/ using document root <none> and password file <none>
I20260929 18:32:27.763502 2167 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:27.763581 2167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:27.763846 2167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:27.766749 2167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "9534aa8a0aa6424eb97fb94baf868076"
format_stamp: "Formatted at 2026-09-29 18:32:27 on dist-test-slave-qcpv"
server_key: "14941a078824506f90a0511e76bce9ca"
server_key_iv: "2466541f5ecf84d853af71d6eecdb41d"
server_key_version: "encryptionkey@0"
I20260929 18:32:27.767553 2167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "9534aa8a0aa6424eb97fb94baf868076"
format_stamp: "Formatted at 2026-09-29 18:32:27 on dist-test-slave-qcpv"
server_key: "14941a078824506f90a0511e76bce9ca"
server_key_iv: "2466541f5ecf84d853af71d6eecdb41d"
server_key_version: "encryptionkey@0"
I20260929 18:32:27.773195 2167 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 18:32:27.777453 2181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:27.779353 2167 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260929 18:32:27.779531 2167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "9534aa8a0aa6424eb97fb94baf868076"
format_stamp: "Formatted at 2026-09-29 18:32:27 on dist-test-slave-qcpv"
server_key: "14941a078824506f90a0511e76bce9ca"
server_key_iv: "2466541f5ecf84d853af71d6eecdb41d"
server_key_version: "encryptionkey@0"
I20260929 18:32:27.779690 2167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:27.800038 2167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:27.801362 2167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:27.801685 2167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:27.811376 2167 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:35415
I20260929 18:32:27.811486 2233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:35415 every 8 connection(s)
I20260929 18:32:27.812733 2167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:32:27.816406 2234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:27.821169 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 2167
I20260929 18:32:27.821317 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:32:27.821678 29501 external_mini_cluster.cc:1428] Setting key 3ebe302da20e7a45ba8a7b345c96c3e0
I20260929 18:32:27.823205 2234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9534aa8a0aa6424eb97fb94baf868076: Bootstrap starting.
I20260929 18:32:27.826148 2234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9534aa8a0aa6424eb97fb94baf868076: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:27.827194 2234 log.cc:824] T 00000000000000000000000000000000 P 9534aa8a0aa6424eb97fb94baf868076: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:27.829996 2234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9534aa8a0aa6424eb97fb94baf868076: No bootstrap required, opened a new log
I20260929 18:32:27.834628 2237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:27.823765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42312 (local address 127.28.207.126:35415)
0929 18:32:27.824676 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:27.824701 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:27.824754 (+ 53us) server_negotiation.cc:407] Connection header received
0929 18:32:27.825659 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:27.825682 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:27.826223 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:27.826678 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:27.827265 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:27.828653 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:27.829728 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:27.830747 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:27.830980 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:27.832123 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:27.832129 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:27.832146 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:32:27.832687 (+ 541us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:27.832762 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:32:27.832789 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:27.832906 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:27.833123 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 18:32:27.833440 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":159,"threads_started":1}
I20260929 18:32:27.834990 2234 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9534aa8a0aa6424eb97fb94baf868076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9534aa8a0aa6424eb97fb94baf868076" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 35415 } }
I20260929 18:32:27.835253 2234 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9534aa8a0aa6424eb97fb94baf868076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:27.835397 2234 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9534aa8a0aa6424eb97fb94baf868076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9534aa8a0aa6424eb97fb94baf868076, State: Initialized, Role: FOLLOWER
I20260929 18:32:27.836015 2234 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9534aa8a0aa6424eb97fb94baf868076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9534aa8a0aa6424eb97fb94baf868076" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 35415 } }
I20260929 18:32:27.836179 2234 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9534aa8a0aa6424eb97fb94baf868076 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:32:27.836315 2234 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9534aa8a0aa6424eb97fb94baf868076 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:32:27.836434 2234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9534aa8a0aa6424eb97fb94baf868076 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:27.837713 2234 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9534aa8a0aa6424eb97fb94baf868076 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9534aa8a0aa6424eb97fb94baf868076" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 35415 } }
I20260929 18:32:27.838150 2234 leader_election.cc:302] T 00000000000000000000000000000000 P 9534aa8a0aa6424eb97fb94baf868076 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9534aa8a0aa6424eb97fb94baf868076; no voters:
I20260929 18:32:27.838542 2234 leader_election.cc:288] T 00000000000000000000000000000000 P 9534aa8a0aa6424eb97fb94baf868076 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:32:27.838833 2239 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9534aa8a0aa6424eb97fb94baf868076 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:32:27.839072 29501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:32:27.839183 2239 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9534aa8a0aa6424eb97fb94baf868076 [term 1 LEADER]: Becoming Leader. State: Replica: 9534aa8a0aa6424eb97fb94baf868076, State: Running, Role: LEADER
I20260929 18:32:27.839632 2239 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9534aa8a0aa6424eb97fb94baf868076 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9534aa8a0aa6424eb97fb94baf868076" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 35415 } }
I20260929 18:32:27.840119 2234 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9534aa8a0aa6424eb97fb94baf868076 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:32:27.842371 2240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9534aa8a0aa6424eb97fb94baf868076 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9534aa8a0aa6424eb97fb94baf868076" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9534aa8a0aa6424eb97fb94baf868076" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 35415 } } }
I20260929 18:32:27.842435 2241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9534aa8a0aa6424eb97fb94baf868076 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9534aa8a0aa6424eb97fb94baf868076. Latest consensus state: current_term: 1 leader_uuid: "9534aa8a0aa6424eb97fb94baf868076" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9534aa8a0aa6424eb97fb94baf868076" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 35415 } } }
I20260929 18:32:27.842595 2241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9534aa8a0aa6424eb97fb94baf868076 [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:27.842594 2240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9534aa8a0aa6424eb97fb94baf868076 [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:27.843303 2254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:32:27.847260 2254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:32:27.855247 2254 catalog_manager.cc:1347] Generated new cluster ID: e84bdad37bbd414387d3eb64fce9a727
I20260929 18:32:27.855388 2254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:32:27.871546 2254 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:32:27.873081 2254 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:32:27.884536 2254 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9534aa8a0aa6424eb97fb94baf868076: Generated new TSK 0
I20260929 18:32:27.885682 2254 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 18:32:27.927721 2264 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 18:32:27.927904 2264 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 18:32:27.928478 2237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:27.908098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42324 (local address 127.28.207.126:35415)
0929 18:32:27.908270 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:27.908276 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:27.908457 (+ 181us) server_negotiation.cc:407] Connection header received
0929 18:32:27.908716 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:27.908719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:27.908790 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:27.908961 (+ 171us) server_negotiation.cc:226] Negotiated authn=JWT
0929 18:32:27.909362 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:27.910248 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:27.911551 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:27.911918 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:27.912028 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:27.912531 (+ 503us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 18:32:27.927893 (+ 15362us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37899/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 18:32:27.928315 (+ 422us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42324: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37899/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":79}
W20260929 18:32:27.928596 2237 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42324: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37899/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 18:32:27.930074 2263 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:27.907939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:35415 (local address 127.0.0.1:42324)
0929 18:32:27.908340 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:27.908356 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:27.908497 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:27.909011 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:27.909015 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:27.909022 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0929 18:32:27.909177 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:27.909183 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:27.910423 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:27.910429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:27.911361 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:27.911381 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:27.912095 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:27.912098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:27.912205 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:27.912271 (+ 66us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 18:32:27.928295 (+ 16024us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37899/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 18:32:27.928656 (+ 361us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.207.126:35415: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37899/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":156,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.28.207.126:35415: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37899/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 18:32:27.931016 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 2167
2026-09-29T18:32:27Z chronyd exiting
I20260929 18:32:27.970494 29501 test_util.cc:175] -----------------------------------------------
I20260929 18:32:27.970618 29501 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790706718260899-29501-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (375 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 18:32:27.971891 29501 mini_oidc.cc:152] Starting JWKS server
I20260929 18:32:27.972877 29501 webserver.cc:460] Webserver started at https://127.0.0.1:37059/ using document root <none> and password file <none>
I20260929 18:32:27.972965 29501 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 18:32:27.973191 29501 webserver.cc:460] Webserver started at http://127.0.0.1:35739/ using document root <none> and password file <none>
2026-09-29T18:32:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:32:27Z Disabled control of system clock
I20260929 18:32:27.985999 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:38077
--webserver_interface=127.28.207.126
--webserver_port=0
--builtin_ntp_servers=127.28.207.84:41125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:38077
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37059/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706718260899-29501-0/testchainca.cert with env {}
W20260929 18:32:28.137918 2270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:28.138481 2270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:28.138559 2270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:28.145604 2270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:32:28.145711 2270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:28.145728 2270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:28.145750 2270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:32:28.145768 2270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 18:32:28.145787 2270 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 18:32:28.145800 2270 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37059/jwks.json
I20260929 18:32:28.154663 2270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:41125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:38077
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:38077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37059/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706718260899-29501-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:28.156574 2270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:28.157734 2270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:28.169304 2275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:28.169304 2276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:28.169494 2270 server_base.cc:1034] running on GCE node
W20260929 18:32:28.169745 2278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:28.170573 2270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:28.172366 2270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:28.173604 2270 hybrid_clock.cc:648] HybridClock initialized: now 1790706748173568 us; error 63 us; skew 500 ppm
I20260929 18:32:28.176558 2270 webserver.cc:460] Webserver started at http://127.28.207.126:39365/ using document root <none> and password file <none>
I20260929 18:32:28.177351 2270 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:28.177430 2270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:28.177701 2270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:28.180454 2270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "334e71d40249458ebb8805e9ffd8daf7"
format_stamp: "Formatted at 2026-09-29 18:32:28 on dist-test-slave-qcpv"
server_key: "d9821b0217546d327797ab8de2129453"
server_key_iv: "5b72aad2c7ed2575163eb2e8792e6fa7"
server_key_version: "encryptionkey@0"
I20260929 18:32:28.181231 2270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "334e71d40249458ebb8805e9ffd8daf7"
format_stamp: "Formatted at 2026-09-29 18:32:28 on dist-test-slave-qcpv"
server_key: "d9821b0217546d327797ab8de2129453"
server_key_iv: "5b72aad2c7ed2575163eb2e8792e6fa7"
server_key_version: "encryptionkey@0"
I20260929 18:32:28.186143 2270 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 18:32:28.189996 2284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:28.191954 2270 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:32:28.192166 2270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "334e71d40249458ebb8805e9ffd8daf7"
format_stamp: "Formatted at 2026-09-29 18:32:28 on dist-test-slave-qcpv"
server_key: "d9821b0217546d327797ab8de2129453"
server_key_iv: "5b72aad2c7ed2575163eb2e8792e6fa7"
server_key_version: "encryptionkey@0"
I20260929 18:32:28.192334 2270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:28.216471 2270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:28.217402 2270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:28.217635 2270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:28.226783 2270 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:38077
I20260929 18:32:28.226792 2336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:38077 every 8 connection(s)
I20260929 18:32:28.228111 2270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:32:28.231787 2337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:28.236989 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 2270
I20260929 18:32:28.237143 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:32:28.237457 29501 external_mini_cluster.cc:1428] Setting key f3a831283d7e47185dbd81a7c838be79
I20260929 18:32:28.237746 2337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 334e71d40249458ebb8805e9ffd8daf7: Bootstrap starting.
I20260929 18:32:28.240466 2337 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 334e71d40249458ebb8805e9ffd8daf7: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:28.241334 2337 log.cc:824] T 00000000000000000000000000000000 P 334e71d40249458ebb8805e9ffd8daf7: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:28.244051 2337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 334e71d40249458ebb8805e9ffd8daf7: No bootstrap required, opened a new log
I20260929 18:32:28.248068 2337 raft_consensus.cc:348] T 00000000000000000000000000000000 P 334e71d40249458ebb8805e9ffd8daf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "334e71d40249458ebb8805e9ffd8daf7" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 38077 } }
I20260929 18:32:28.248333 2337 raft_consensus.cc:374] T 00000000000000000000000000000000 P 334e71d40249458ebb8805e9ffd8daf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:28.248373 2337 raft_consensus.cc:729] T 00000000000000000000000000000000 P 334e71d40249458ebb8805e9ffd8daf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 334e71d40249458ebb8805e9ffd8daf7, State: Initialized, Role: FOLLOWER
I20260929 18:32:28.248973 2337 consensus_queue.cc:260] T 00000000000000000000000000000000 P 334e71d40249458ebb8805e9ffd8daf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "334e71d40249458ebb8805e9ffd8daf7" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 38077 } }
I20260929 18:32:28.249166 2337 raft_consensus.cc:388] T 00000000000000000000000000000000 P 334e71d40249458ebb8805e9ffd8daf7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:32:28.249258 2337 raft_consensus.cc:482] T 00000000000000000000000000000000 P 334e71d40249458ebb8805e9ffd8daf7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:32:28.249377 2337 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 334e71d40249458ebb8805e9ffd8daf7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:28.250267 2340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:28.239706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33852 (local address 127.28.207.126:38077)
0929 18:32:28.240876 (+ 1170us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:28.240903 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:28.240962 (+ 59us) server_negotiation.cc:407] Connection header received
0929 18:32:28.242169 (+ 1207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:28.242203 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:28.242754 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:28.243268 (+ 514us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:28.243815 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:28.244967 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:28.245973 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:28.246575 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:28.246786 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:28.247850 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:28.247857 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:28.247875 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:32:28.248438 (+ 563us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:28.248541 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:32:28.248574 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:28.248684 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:28.248817 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 18:32:28.249193 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":165,"threads_started":1}
I20260929 18:32:28.250546 2337 raft_consensus.cc:504] T 00000000000000000000000000000000 P 334e71d40249458ebb8805e9ffd8daf7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "334e71d40249458ebb8805e9ffd8daf7" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 38077 } }
I20260929 18:32:28.251039 2337 leader_election.cc:302] T 00000000000000000000000000000000 P 334e71d40249458ebb8805e9ffd8daf7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 334e71d40249458ebb8805e9ffd8daf7; no voters:
I20260929 18:32:28.251442 2337 leader_election.cc:288] T 00000000000000000000000000000000 P 334e71d40249458ebb8805e9ffd8daf7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:32:28.251538 2342 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 334e71d40249458ebb8805e9ffd8daf7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:32:28.251745 2342 raft_consensus.cc:686] T 00000000000000000000000000000000 P 334e71d40249458ebb8805e9ffd8daf7 [term 1 LEADER]: Becoming Leader. State: Replica: 334e71d40249458ebb8805e9ffd8daf7, State: Running, Role: LEADER
I20260929 18:32:28.252100 2342 consensus_queue.cc:237] T 00000000000000000000000000000000 P 334e71d40249458ebb8805e9ffd8daf7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "334e71d40249458ebb8805e9ffd8daf7" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 38077 } }
I20260929 18:32:28.252938 2337 sys_catalog.cc:564] T 00000000000000000000000000000000 P 334e71d40249458ebb8805e9ffd8daf7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:32:28.254560 29501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:32:28.254560 2343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 334e71d40249458ebb8805e9ffd8daf7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "334e71d40249458ebb8805e9ffd8daf7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "334e71d40249458ebb8805e9ffd8daf7" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 38077 } } }
I20260929 18:32:28.254760 2343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 334e71d40249458ebb8805e9ffd8daf7 [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:28.255751 2344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 334e71d40249458ebb8805e9ffd8daf7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 334e71d40249458ebb8805e9ffd8daf7. Latest consensus state: current_term: 1 leader_uuid: "334e71d40249458ebb8805e9ffd8daf7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "334e71d40249458ebb8805e9ffd8daf7" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 38077 } } }
I20260929 18:32:28.255901 2344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 334e71d40249458ebb8805e9ffd8daf7 [sys.catalog]: This master's current role is: LEADER
W20260929 18:32:28.258386 2357 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 334e71d40249458ebb8805e9ffd8daf7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:32:28.258466 2357 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:32:28.258551 2354 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:32:28.259380 2354 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:32:28.267518 2354 catalog_manager.cc:1347] Generated new cluster ID: f13de129cfc24496b0bc40688a3d15a9
I20260929 18:32:28.267671 2354 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:32:28.281544 2354 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:32:28.282575 2354 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:32:28.288715 2354 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 334e71d40249458ebb8805e9ffd8daf7: Generated new TSK 0
I20260929 18:32:28.289500 2354 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 18:32:28.320719 2367 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 18:32:28.320875 2367 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260929 18:32:28.321915 2366 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:28.303373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:38077 (local address 127.0.0.1:33864)
0929 18:32:28.303685 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:28.303698 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:28.303797 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:28.304313 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:28.304316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:28.304325 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0929 18:32:28.304440 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:28.304445 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:28.305683 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:28.305686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:28.306436 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:28.306445 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:28.307357 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:28.307360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:28.307440 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:28.307467 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 18:32:28.321525 (+ 14058us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37059/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 18:32:28.321755 (+ 230us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.207.126:38077: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37059/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":117,"threads_started":1}
I20260929 18:32:28.321892 2340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:28.303535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33864 (local address 127.28.207.126:38077)
0929 18:32:28.303716 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:28.303722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:28.303751 (+ 29us) server_negotiation.cc:407] Connection header received
0929 18:32:28.304005 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:28.304009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:28.304089 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:28.304266 (+ 177us) server_negotiation.cc:226] Negotiated authn=JWT
0929 18:32:28.304598 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:28.305525 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:28.306668 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:28.307180 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:28.307317 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:28.307709 (+ 392us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 18:32:28.321212 (+ 13503us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37059/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 18:32:28.321590 (+ 378us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33864: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37059/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":72}
W20260929 18:32:28.322022 2340 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33864: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37059/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260929 18:32:28.322481 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 2270
2026-09-29T18:32:28Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (392 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 18:32:28.364694 29501 mini_oidc.cc:152] Starting JWKS server
I20260929 18:32:28.365890 29501 webserver.cc:460] Webserver started at https://127.0.0.1:43071/ using document root <none> and password file <none>
I20260929 18:32:28.366055 29501 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 18:32:28.366412 29501 webserver.cc:460] Webserver started at http://127.0.0.1:42061/ using document root <none> and password file <none>
2026-09-29T18:32:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:32:28Z Disabled control of system clock
I20260929 18:32:28.378244 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:45571
--webserver_interface=127.28.207.126
--webserver_port=0
--builtin_ntp_servers=127.28.207.84:42155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:45571
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43071/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 18:32:28.523996 2373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:28.524389 2373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:28.524434 2373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:28.528806 2373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:32:28.528888 2373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:28.528904 2373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:28.528923 2373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:32:28.528939 2373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 18:32:28.528955 2373 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 18:32:28.528970 2373 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43071/jwks.json
I20260929 18:32:28.534132 2373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:42155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:45571
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:45571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43071/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:28.535604 2373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:28.537469 2373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:28.548044 2379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:28.548077 2378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:28.548069 2381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:28.549445 2373 server_base.cc:1034] running on GCE node
I20260929 18:32:28.550184 2373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:28.551548 2373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:28.552740 2373 hybrid_clock.cc:648] HybridClock initialized: now 1790706748552727 us; error 44 us; skew 500 ppm
I20260929 18:32:28.555523 2373 webserver.cc:460] Webserver started at http://127.28.207.126:33113/ using document root <none> and password file <none>
I20260929 18:32:28.556216 2373 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:28.556294 2373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:28.556551 2373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:28.558911 2373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "863fb863a6fa4843a929fb1ca8090f7d"
format_stamp: "Formatted at 2026-09-29 18:32:28 on dist-test-slave-qcpv"
server_key: "e0e25ed5be45a65237adf43b48afe4f1"
server_key_iv: "4ab4cc9fadc72d9efb4ee83d6a04564e"
server_key_version: "encryptionkey@0"
I20260929 18:32:28.559557 2373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "863fb863a6fa4843a929fb1ca8090f7d"
format_stamp: "Formatted at 2026-09-29 18:32:28 on dist-test-slave-qcpv"
server_key: "e0e25ed5be45a65237adf43b48afe4f1"
server_key_iv: "4ab4cc9fadc72d9efb4ee83d6a04564e"
server_key_version: "encryptionkey@0"
I20260929 18:32:28.565951 2373 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260929 18:32:28.569118 2387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:28.571203 2373 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 18:32:28.571385 2373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "863fb863a6fa4843a929fb1ca8090f7d"
format_stamp: "Formatted at 2026-09-29 18:32:28 on dist-test-slave-qcpv"
server_key: "e0e25ed5be45a65237adf43b48afe4f1"
server_key_iv: "4ab4cc9fadc72d9efb4ee83d6a04564e"
server_key_version: "encryptionkey@0"
I20260929 18:32:28.571542 2373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:28.584848 2373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:28.585619 2373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:28.585832 2373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:28.594565 2373 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:45571
I20260929 18:32:28.594594 2439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:45571 every 8 connection(s)
I20260929 18:32:28.595709 2373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:32:28.597461 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 2373
I20260929 18:32:28.597616 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:32:28.597980 29501 external_mini_cluster.cc:1428] Setting key cac874ff946f8c781d87de116285cedb
I20260929 18:32:28.601123 2440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:28.608770 2440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 863fb863a6fa4843a929fb1ca8090f7d: Bootstrap starting.
I20260929 18:32:28.609856 2443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:28.600202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37690 (local address 127.28.207.126:45571)
0929 18:32:28.601175 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:28.601226 (+ 51us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:28.601291 (+ 65us) server_negotiation.cc:407] Connection header received
0929 18:32:28.602012 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:28.602031 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:28.602651 (+ 620us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:28.603107 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:28.603675 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:28.604788 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:28.605794 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:28.606343 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:28.606568 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:28.607772 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:28.607777 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:28.607792 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:32:28.608255 (+ 463us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:28.608322 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:32:28.608343 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:28.608498 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:28.608631 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 18:32:28.608950 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":220,"threads_started":1}
I20260929 18:32:28.611999 2440 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 863fb863a6fa4843a929fb1ca8090f7d: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:28.612998 2440 log.cc:824] T 00000000000000000000000000000000 P 863fb863a6fa4843a929fb1ca8090f7d: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:28.613240 29501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:32:28.615696 2440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 863fb863a6fa4843a929fb1ca8090f7d: No bootstrap required, opened a new log
I20260929 18:32:28.620056 2440 raft_consensus.cc:348] T 00000000000000000000000000000000 P 863fb863a6fa4843a929fb1ca8090f7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "863fb863a6fa4843a929fb1ca8090f7d" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 45571 } }
I20260929 18:32:28.620371 2440 raft_consensus.cc:374] T 00000000000000000000000000000000 P 863fb863a6fa4843a929fb1ca8090f7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:28.620468 2440 raft_consensus.cc:729] T 00000000000000000000000000000000 P 863fb863a6fa4843a929fb1ca8090f7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 863fb863a6fa4843a929fb1ca8090f7d, State: Initialized, Role: FOLLOWER
I20260929 18:32:28.620986 2440 consensus_queue.cc:260] T 00000000000000000000000000000000 P 863fb863a6fa4843a929fb1ca8090f7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "863fb863a6fa4843a929fb1ca8090f7d" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 45571 } }
I20260929 18:32:28.621158 2440 raft_consensus.cc:388] T 00000000000000000000000000000000 P 863fb863a6fa4843a929fb1ca8090f7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:32:28.621250 2440 raft_consensus.cc:482] T 00000000000000000000000000000000 P 863fb863a6fa4843a929fb1ca8090f7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:32:28.621380 2440 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 863fb863a6fa4843a929fb1ca8090f7d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:28.622560 2440 raft_consensus.cc:504] T 00000000000000000000000000000000 P 863fb863a6fa4843a929fb1ca8090f7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "863fb863a6fa4843a929fb1ca8090f7d" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 45571 } }
I20260929 18:32:28.623059 2440 leader_election.cc:302] T 00000000000000000000000000000000 P 863fb863a6fa4843a929fb1ca8090f7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 863fb863a6fa4843a929fb1ca8090f7d; no voters:
I20260929 18:32:28.623471 2440 leader_election.cc:288] T 00000000000000000000000000000000 P 863fb863a6fa4843a929fb1ca8090f7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:32:28.623715 2445 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 863fb863a6fa4843a929fb1ca8090f7d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:32:28.624073 2445 raft_consensus.cc:686] T 00000000000000000000000000000000 P 863fb863a6fa4843a929fb1ca8090f7d [term 1 LEADER]: Becoming Leader. State: Replica: 863fb863a6fa4843a929fb1ca8090f7d, State: Running, Role: LEADER
I20260929 18:32:28.624487 2445 consensus_queue.cc:237] T 00000000000000000000000000000000 P 863fb863a6fa4843a929fb1ca8090f7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "863fb863a6fa4843a929fb1ca8090f7d" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 45571 } }
I20260929 18:32:28.625134 2440 sys_catalog.cc:564] T 00000000000000000000000000000000 P 863fb863a6fa4843a929fb1ca8090f7d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:32:28.626770 2447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 863fb863a6fa4843a929fb1ca8090f7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 863fb863a6fa4843a929fb1ca8090f7d. Latest consensus state: current_term: 1 leader_uuid: "863fb863a6fa4843a929fb1ca8090f7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "863fb863a6fa4843a929fb1ca8090f7d" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 45571 } } }
I20260929 18:32:28.627403 2447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 863fb863a6fa4843a929fb1ca8090f7d [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:28.628270 2446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 863fb863a6fa4843a929fb1ca8090f7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "863fb863a6fa4843a929fb1ca8090f7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "863fb863a6fa4843a929fb1ca8090f7d" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 45571 } } }
I20260929 18:32:28.628476 2455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:32:28.628711 2446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 863fb863a6fa4843a929fb1ca8090f7d [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:28.631471 2455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:32:28.637939 2455 catalog_manager.cc:1347] Generated new cluster ID: a6141ecea0a14dc59207eacf19e3e8a2
I20260929 18:32:28.638046 2455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:32:28.647535 2455 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:32:28.648584 2455 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:32:28.657531 2455 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 863fb863a6fa4843a929fb1ca8090f7d: Generated new TSK 0
I20260929 18:32:28.658522 2455 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:32:28.681128 2443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:28.674530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37696 (local address 127.28.207.126:45571)
0929 18:32:28.674688 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:28.674693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:28.674955 (+ 262us) server_negotiation.cc:407] Connection header received
0929 18:32:28.675127 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:28.675130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:28.675202 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:28.675345 (+ 143us) server_negotiation.cc:226] Negotiated authn=JWT
0929 18:32:28.675692 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:28.676530 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:28.677961 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:28.678218 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:28.678333 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:28.678783 (+ 450us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 18:32:28.680655 (+ 1872us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43071/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 18:32:28.680934 (+ 279us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37696: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43071/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":58}
W20260929 18:32:28.681221 2443 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37696: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43071/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 18:32:28.683126 2469 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:28.674418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45571 (local address 127.0.0.1:37696)
0929 18:32:28.674771 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:28.674787 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:28.674963 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:28.675366 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:28.675369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:28.675378 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0929 18:32:28.675513 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:28.675520 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:28.676674 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:28.676679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:28.677748 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:28.677761 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:28.678382 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:28.678386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:28.678482 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:28.678519 (+ 37us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 18:32:28.680877 (+ 2358us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43071/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 18:32:28.682889 (+ 2012us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.207.126:45571: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43071/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":1714,"thread_start_us":119,"threads_started":1}
I20260929 18:32:28.684369 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 2373
E20260929 18:32:28.692276 2470 webserver.cc:533] Webserver: sslize failed:
E20260929 18:32:28.692495 2470 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-29T18:32:28Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (360 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T18:32:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:32:28Z Disabled control of system clock
I20260929 18:32:28.736070 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:43327
--webserver_interface=127.28.207.126
--webserver_port=0
--builtin_ntp_servers=127.28.207.84:36567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:43327
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706718260899-29501-0/testchainca.cert with env {}
W20260929 18:32:28.860086 2474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:28.860440 2474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:28.860488 2474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:28.864675 2474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:32:28.864773 2474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:28.864792 2474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:28.864812 2474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:32:28.864830 2474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 18:32:28.864851 2474 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 18:32:28.864867 2474 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 18:32:28.870652 2474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:36567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:43327
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:43327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706718260899-29501-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:28.871881 2474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:28.872927 2474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:28.882823 2479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:28.883040 2482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:28.882823 2480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:28.883210 2474 server_base.cc:1034] running on GCE node
I20260929 18:32:28.884038 2474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:28.885234 2474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:28.886438 2474 hybrid_clock.cc:648] HybridClock initialized: now 1790706748886399 us; error 66 us; skew 500 ppm
I20260929 18:32:28.894618 2474 webserver.cc:460] Webserver started at http://127.28.207.126:40611/ using document root <none> and password file <none>
I20260929 18:32:28.895361 2474 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:28.895445 2474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:28.895630 2474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:28.897766 2474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "5f7bdd754be1437fab7fde3052234c01"
format_stamp: "Formatted at 2026-09-29 18:32:28 on dist-test-slave-qcpv"
server_key: "bf3a374b02572bd74d145cfdca7fdf52"
server_key_iv: "26579d6d8f19b48f9306052d796a53ed"
server_key_version: "encryptionkey@0"
I20260929 18:32:28.898319 2474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "5f7bdd754be1437fab7fde3052234c01"
format_stamp: "Formatted at 2026-09-29 18:32:28 on dist-test-slave-qcpv"
server_key: "bf3a374b02572bd74d145cfdca7fdf52"
server_key_iv: "26579d6d8f19b48f9306052d796a53ed"
server_key_version: "encryptionkey@0"
I20260929 18:32:28.903033 2474 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 18:32:28.906522 2488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:28.908394 2474 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:32:28.908553 2474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "5f7bdd754be1437fab7fde3052234c01"
format_stamp: "Formatted at 2026-09-29 18:32:28 on dist-test-slave-qcpv"
server_key: "bf3a374b02572bd74d145cfdca7fdf52"
server_key_iv: "26579d6d8f19b48f9306052d796a53ed"
server_key_version: "encryptionkey@0"
I20260929 18:32:28.908672 2474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:28.939160 2474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:28.939927 2474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:28.940117 2474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:28.948475 2474 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:43327
I20260929 18:32:28.948494 2540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:43327 every 8 connection(s)
I20260929 18:32:28.949539 2474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:32:28.953737 2541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:28.954674 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 2474
I20260929 18:32:28.954789 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:32:28.955410 29501 external_mini_cluster.cc:1428] Setting key 95101d61287d01fd673e76d7e055f578
I20260929 18:32:28.962370 2541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f7bdd754be1437fab7fde3052234c01: Bootstrap starting.
I20260929 18:32:28.966418 2541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f7bdd754be1437fab7fde3052234c01: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:28.967552 2541 log.cc:824] T 00000000000000000000000000000000 P 5f7bdd754be1437fab7fde3052234c01: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:28.967605 2544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:28.957071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41400 (local address 127.28.207.126:43327)
0929 18:32:28.957833 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:28.957849 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:28.957894 (+ 45us) server_negotiation.cc:407] Connection header received
0929 18:32:28.958751 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:28.958772 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:28.959317 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:28.959668 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:28.960301 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:28.961400 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:28.962939 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:28.963582 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:28.963777 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:28.965142 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:28.965148 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:28.965173 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:32:28.965746 (+ 573us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:28.965827 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:32:28.965853 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:28.965973 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:28.966242 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 18:32:28.966648 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":171,"threads_started":1}
I20260929 18:32:28.971124 2541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f7bdd754be1437fab7fde3052234c01: No bootstrap required, opened a new log
I20260929 18:32:28.972520 29501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:32:28.977110 2541 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f7bdd754be1437fab7fde3052234c01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7bdd754be1437fab7fde3052234c01" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43327 } }
I20260929 18:32:28.977581 2541 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f7bdd754be1437fab7fde3052234c01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:28.977689 2541 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f7bdd754be1437fab7fde3052234c01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f7bdd754be1437fab7fde3052234c01, State: Initialized, Role: FOLLOWER
I20260929 18:32:28.978345 2541 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f7bdd754be1437fab7fde3052234c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7bdd754be1437fab7fde3052234c01" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43327 } }
I20260929 18:32:28.978539 2541 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f7bdd754be1437fab7fde3052234c01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:32:28.978638 2541 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f7bdd754be1437fab7fde3052234c01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:32:28.978780 2541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f7bdd754be1437fab7fde3052234c01 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:28.980082 2541 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f7bdd754be1437fab7fde3052234c01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7bdd754be1437fab7fde3052234c01" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43327 } }
I20260929 18:32:28.980543 2541 leader_election.cc:302] T 00000000000000000000000000000000 P 5f7bdd754be1437fab7fde3052234c01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f7bdd754be1437fab7fde3052234c01; no voters:
I20260929 18:32:28.980967 2541 leader_election.cc:288] T 00000000000000000000000000000000 P 5f7bdd754be1437fab7fde3052234c01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:32:28.981179 2546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f7bdd754be1437fab7fde3052234c01 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:32:28.981546 2546 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f7bdd754be1437fab7fde3052234c01 [term 1 LEADER]: Becoming Leader. State: Replica: 5f7bdd754be1437fab7fde3052234c01, State: Running, Role: LEADER
I20260929 18:32:28.981967 2546 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f7bdd754be1437fab7fde3052234c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7bdd754be1437fab7fde3052234c01" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43327 } }
I20260929 18:32:28.982681 2541 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f7bdd754be1437fab7fde3052234c01 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:32:28.984127 2547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f7bdd754be1437fab7fde3052234c01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f7bdd754be1437fab7fde3052234c01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7bdd754be1437fab7fde3052234c01" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43327 } } }
I20260929 18:32:28.984331 2547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f7bdd754be1437fab7fde3052234c01 [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:28.984858 2555 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:32:28.984128 2548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f7bdd754be1437fab7fde3052234c01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f7bdd754be1437fab7fde3052234c01. Latest consensus state: current_term: 1 leader_uuid: "5f7bdd754be1437fab7fde3052234c01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7bdd754be1437fab7fde3052234c01" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43327 } } }
I20260929 18:32:28.985131 2548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f7bdd754be1437fab7fde3052234c01 [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:28.989651 2555 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:32:28.997753 2555 catalog_manager.cc:1347] Generated new cluster ID: 900aa92d40ba496fa369958d7cf9411f
I20260929 18:32:28.997884 2555 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:32:29.008831 2555 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:32:29.010300 2555 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:32:29.020035 2555 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f7bdd754be1437fab7fde3052234c01: Generated new TSK 0
I20260929 18:32:29.021008 2555 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:32:29.096016 2544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:29.038450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41402 (local address 127.28.207.126:43327)
0929 18:32:29.038649 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:29.038654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:29.039015 (+ 361us) server_negotiation.cc:407] Connection header received
0929 18:32:29.039142 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:29.039145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:29.039213 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:29.039341 (+ 128us) server_negotiation.cc:226] Negotiated authn=JWT
0929 18:32:29.039648 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:29.040293 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:29.041685 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:29.041962 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:29.042074 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:29.042546 (+ 472us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 18:32:29.095466 (+ 52920us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 18:32:29.095855 (+ 389us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41402: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":78}
W20260929 18:32:29.096145 2544 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41402: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 18:32:29.096679 2570 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:29.038328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43327 (local address 127.0.0.1:41402)
0929 18:32:29.038771 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:29.038792 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:29.038988 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:29.039372 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:29.039376 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:29.039386 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0929 18:32:29.039506 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:29.039513 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:29.040420 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:29.040424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:29.041495 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:29.041506 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:29.042124 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:29.042128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:29.042228 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:29.042262 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 18:32:29.095830 (+ 53568us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 18:32:29.096107 (+ 277us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.28.207.126:43327: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":172,"threads_started":1}
I20260929 18:32:29.097658 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 2474
2026-09-29T18:32:29Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (394 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (134 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (163 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:32:29 dist-test-slave-qcpv krb5kdc[2582](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:32:29 dist-test-slave-qcpv krb5kdc[2582](info): set up 2 sockets
Sep 29 18:32:29 dist-test-slave-qcpv krb5kdc[2582](info): commencing operation
krb5kdc: starting...
W20260929 18:32:30.479538 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.008s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:32:30 dist-test-slave-qcpv krb5kdc[2582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:32:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:32:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.207.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
I20260929 18:32:30.619454 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:42681
--webserver_interface=127.28.207.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:44105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:42681
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:30.741454 2595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:30.741796 2595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:30.741843 2595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:30.746148 2595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:32:30.746258 2595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:30.746281 2595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:30.746301 2595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:32:30.746326 2595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:32:30.752185 2595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:44105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:42681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:42681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:30.753398 2595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:30.754490 2595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:30.763975 2601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:30.763986 2600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:30.763986 2603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:30.764501 2595 server_base.cc:1034] running on GCE node
I20260929 18:32:30.765290 2595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:30.766825 2595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:30.768208 2595 hybrid_clock.cc:648] HybridClock initialized: now 1790706750768147 us; error 91 us; skew 500 ppm
Sep 29 18:32:30 dist-test-slave-qcpv krb5kdc[2582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706750, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:30.771768 2595 init.cc:377] Logged in from keytab as kudu/127.28.207.126@KRBTEST.COM (short username kudu)
I20260929 18:32:30.772939 2595 webserver.cc:460] Webserver started at http://127.28.207.126:36057/ using document root <none> and password file <none>
I20260929 18:32:30.773751 2595 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:30.773833 2595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:30.774046 2595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:30.776758 2595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "3357292f973c438188eb397535f624ac"
format_stamp: "Formatted at 2026-09-29 18:32:30 on dist-test-slave-qcpv"
server_key: "c10ebf34e02997b69c93adc2bf4d3f05"
server_key_iv: "0454767af81b2eda086cf5e7dae20c77"
server_key_version: "encryptionkey@0"
I20260929 18:32:30.777630 2595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "3357292f973c438188eb397535f624ac"
format_stamp: "Formatted at 2026-09-29 18:32:30 on dist-test-slave-qcpv"
server_key: "c10ebf34e02997b69c93adc2bf4d3f05"
server_key_iv: "0454767af81b2eda086cf5e7dae20c77"
server_key_version: "encryptionkey@0"
I20260929 18:32:30.782711 2595 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 18:32:30.786398 2610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:30.788360 2595 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:32:30.788611 2595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "3357292f973c438188eb397535f624ac"
format_stamp: "Formatted at 2026-09-29 18:32:30 on dist-test-slave-qcpv"
server_key: "c10ebf34e02997b69c93adc2bf4d3f05"
server_key_iv: "0454767af81b2eda086cf5e7dae20c77"
server_key_version: "encryptionkey@0"
I20260929 18:32:30.788755 2595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:30.810245 2595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:30.814136 2595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:30.814373 2595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:30.824536 2595 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:42681
I20260929 18:32:30.824626 2662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:42681 every 8 connection(s)
I20260929 18:32:30.825838 2595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:32:30.828275 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 2595
I20260929 18:32:30.828449 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:32:30.828789 29501 external_mini_cluster.cc:1428] Setting key eb24951eca03bd9cb6b987e89567152f
I20260929 18:32:30.829548 2663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:30.837806 2663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3357292f973c438188eb397535f624ac: Bootstrap starting.
Sep 29 18:32:30 dist-test-slave-qcpv krb5kdc[2582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:32:30.840986 2663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3357292f973c438188eb397535f624ac: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:30.841936 2663 log.cc:824] T 00000000000000000000000000000000 P 3357292f973c438188eb397535f624ac: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:30.844162 2663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3357292f973c438188eb397535f624ac: No bootstrap required, opened a new log
I20260929 18:32:30.848007 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:30.830566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56708 (local address 127.28.207.126:42681)
0929 18:32:30.831115 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:30.831145 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:30.831193 (+ 48us) server_negotiation.cc:407] Connection header received
0929 18:32:30.831996 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:30.832016 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:30.832446 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:30.832822 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:30.834015 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:30.835296 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:30.836757 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:30.837384 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:30.837548 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:30.841215 (+ 3667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:30.841223 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:30.841244 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:30.841291 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:30.843942 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:30.844650 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:30.844663 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:30.844669 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:30.844897 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:30.845280 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:30.845285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:30.845287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:30.845884 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:30.846143 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:30.846486 (+ 343us) server_negotiation.cc:299] Negotiation successful
0929 18:32:30.846965 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":212,"threads_started":1}
I20260929 18:32:30.848446 2663 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3357292f973c438188eb397535f624ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3357292f973c438188eb397535f624ac" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 42681 } }
I20260929 18:32:30.848742 2663 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3357292f973c438188eb397535f624ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:30.848812 2663 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3357292f973c438188eb397535f624ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3357292f973c438188eb397535f624ac, State: Initialized, Role: FOLLOWER
I20260929 18:32:30.849354 2663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3357292f973c438188eb397535f624ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3357292f973c438188eb397535f624ac" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 42681 } }
I20260929 18:32:30.849531 2663 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3357292f973c438188eb397535f624ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:32:30.849623 2663 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3357292f973c438188eb397535f624ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:32:30.849750 2663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3357292f973c438188eb397535f624ac [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:30.850777 2663 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3357292f973c438188eb397535f624ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3357292f973c438188eb397535f624ac" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 42681 } }
I20260929 18:32:30.851233 2663 leader_election.cc:302] T 00000000000000000000000000000000 P 3357292f973c438188eb397535f624ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3357292f973c438188eb397535f624ac; no voters:
I20260929 18:32:30.851606 2663 leader_election.cc:288] T 00000000000000000000000000000000 P 3357292f973c438188eb397535f624ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:32:30.851876 2668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3357292f973c438188eb397535f624ac [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:32:30.852263 2668 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3357292f973c438188eb397535f624ac [term 1 LEADER]: Becoming Leader. State: Replica: 3357292f973c438188eb397535f624ac, State: Running, Role: LEADER
I20260929 18:32:30.852671 2668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3357292f973c438188eb397535f624ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3357292f973c438188eb397535f624ac" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 42681 } }
I20260929 18:32:30.853014 2663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3357292f973c438188eb397535f624ac [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:32:30.854748 2669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3357292f973c438188eb397535f624ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3357292f973c438188eb397535f624ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3357292f973c438188eb397535f624ac" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 42681 } } }
I20260929 18:32:30.854977 2669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3357292f973c438188eb397535f624ac [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:30.855387 2670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3357292f973c438188eb397535f624ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3357292f973c438188eb397535f624ac. Latest consensus state: current_term: 1 leader_uuid: "3357292f973c438188eb397535f624ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3357292f973c438188eb397535f624ac" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 42681 } } }
I20260929 18:32:30.855515 2670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3357292f973c438188eb397535f624ac [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:30.855902 2678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:32:30.860673 2678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:32:30.868245 2678 catalog_manager.cc:1347] Generated new cluster ID: 718c92c8180d4e77a00fdcc95173285b
I20260929 18:32:30.868368 2678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:32:30.878410 2678 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:32:30.879860 2678 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:32:30.886000 2678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3357292f973c438188eb397535f624ac: Generated new TSK 0
I20260929 18:32:30.887012 2678 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.207.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab.
Entry for principal kudu/127.28.207.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab.
I20260929 18:32:30.932022 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.65:0
--local_ip_for_outbound_sockets=127.28.207.65
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:42681
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:44105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:31.090010 2689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:31.090453 2689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:31.090549 2689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:31.097606 2689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:31.097729 2689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:31.097841 2689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.65
I20260929 18:32:31.105919 2689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:44105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:42681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.65
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:31.107573 2689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:31.108872 2689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:31.126174 2695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:31.126174 2697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:31.126281 2689 server_base.cc:1034] running on GCE node
W20260929 18:32:31.126194 2694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:31.127219 2689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:31.128072 2689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:31.129334 2689 hybrid_clock.cc:648] HybridClock initialized: now 1790706751129312 us; error 47 us; skew 500 ppm
Sep 29 18:32:31 dist-test-slave-qcpv krb5kdc[2582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706751, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:31.133888 2689 init.cc:377] Logged in from keytab as kudu/127.28.207.65@KRBTEST.COM (short username kudu)
I20260929 18:32:31.135198 2689 webserver.cc:460] Webserver started at http://127.28.207.65:37481/ using document root <none> and password file <none>
I20260929 18:32:31.136184 2689 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:31.136281 2689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:31.136520 2689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:31.139508 2689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-0/data/instance:
uuid: "706054a3e41146e788cfefef74c961da"
format_stamp: "Formatted at 2026-09-29 18:32:31 on dist-test-slave-qcpv"
server_key: "8671118919bf8bd423e251ab8e36b4e2"
server_key_iv: "d2665e24431af2c522cdb6d622f39323"
server_key_version: "encryptionkey@0"
I20260929 18:32:31.140168 2689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance:
uuid: "706054a3e41146e788cfefef74c961da"
format_stamp: "Formatted at 2026-09-29 18:32:31 on dist-test-slave-qcpv"
server_key: "8671118919bf8bd423e251ab8e36b4e2"
server_key_iv: "d2665e24431af2c522cdb6d622f39323"
server_key_version: "encryptionkey@0"
I20260929 18:32:31.145419 2689 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 18:32:31.149154 2704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:31.151137 2689 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:32:31.151336 2689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-0/wal
uuid: "706054a3e41146e788cfefef74c961da"
format_stamp: "Formatted at 2026-09-29 18:32:31 on dist-test-slave-qcpv"
server_key: "8671118919bf8bd423e251ab8e36b4e2"
server_key_iv: "d2665e24431af2c522cdb6d622f39323"
server_key_version: "encryptionkey@0"
I20260929 18:32:31.151510 2689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:31.173357 2689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:31.176939 2689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:31.177166 2689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:31.177802 2689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:32:31.178694 2689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:31.178766 2689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:31.178898 2689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:31.178956 2689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:31.190287 2689 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.65:34831
I20260929 18:32:31.190402 2817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.65:34831 every 8 connection(s)
I20260929 18:32:31.191464 2689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
I20260929 18:32:31.192052 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 2689
I20260929 18:32:31.192196 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance
I20260929 18:32:31.192569 29501 external_mini_cluster.cc:1428] Setting key ac5b3ba33395a1fe09c87b81a41c9ec8
WARNING: no policy specified for kudu/127.28.207.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.66@KRBTEST.COM" created.
Sep 29 18:32:31 dist-test-slave-qcpv krb5kdc[2582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706751, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:32:31.213737 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:31.193632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:57879 (local address 127.28.207.126:42681)
0929 18:32:31.193832 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:31.193838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:31.195157 (+ 1319us) server_negotiation.cc:407] Connection header received
0929 18:32:31.197509 (+ 2352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:31.197513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:31.197717 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:31.197837 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:31.199682 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:31.200590 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:31.201978 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:31.202416 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:31.202564 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:31.208628 (+ 6064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:31.208636 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:31.208644 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:31.208697 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:31.211284 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:31.211930 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:31.211935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:31.211936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:31.212045 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:31.212421 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:31.212429 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:31.212430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:31.212662 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:31.212801 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:31.213370 (+ 569us) server_negotiation.cc:299] Negotiation successful
0929 18:32:31.213542 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 18:32:31.214633 2820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:31.194243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:42681 (local address 127.28.207.65:57879)
0929 18:32:31.194900 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:31.194999 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:31.197211 (+ 2212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:31.198276 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:31.198285 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:31.198752 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:31.199519 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:31.199538 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:31.200796 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:31.200801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:31.201807 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:31.201820 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:31.202593 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:31.202596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:31.202775 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:31.203541 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:31.203558 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:31.208346 (+ 4788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:31.211440 (+ 3094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:31.211445 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:31.211460 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:31.211762 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:31.212157 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:31.212160 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:31.212162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:31.212312 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:31.212791 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:31.212798 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:31.213069 (+ 271us) client_negotiation.cc:769] Sending connection context
0929 18:32:31.213313 (+ 244us) client_negotiation.cc:240] Negotiation successful
0929 18:32:31.213663 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":125,"threads_started":1}
I20260929 18:32:31.216509 2818 heartbeater.cc:347] Connected to a master server at 127.28.207.126:42681
I20260929 18:32:31.216804 2818 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:31.217343 2818 heartbeater.cc:511] Master 127.28.207.126:42681 requested a full tablet report, sending...
I20260929 18:32:31.219172 2627 ts_manager.cc:194] Registered new tserver with Master: 706054a3e41146e788cfefef74c961da (127.28.207.65:34831)
I20260929 18:32:31.220410 2627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.65@KRBTEST.COM'} at 127.28.207.65:57879
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab.
Entry for principal kudu/127.28.207.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab.
I20260929 18:32:31.231875 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.66:0
--local_ip_for_outbound_sockets=127.28.207.66
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:42681
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:44105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:31.384227 2823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:31.384536 2823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:31.384586 2823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:31.390201 2823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:31.390372 2823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:31.390467 2823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.66
I20260929 18:32:31.398300 2823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:44105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:42681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.66
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:31.400250 2823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:31.401974 2823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:31.416030 2831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:31.416031 2828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:31.416446 2823 server_base.cc:1034] running on GCE node
W20260929 18:32:31.416030 2829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:31.417142 2823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:31.417907 2823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:31.419122 2823 hybrid_clock.cc:648] HybridClock initialized: now 1790706751419083 us; error 65 us; skew 500 ppm
Sep 29 18:32:31 dist-test-slave-qcpv krb5kdc[2582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706751, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:31.422416 2823 init.cc:377] Logged in from keytab as kudu/127.28.207.66@KRBTEST.COM (short username kudu)
I20260929 18:32:31.423493 2823 webserver.cc:460] Webserver started at http://127.28.207.66:43557/ using document root <none> and password file <none>
I20260929 18:32:31.424126 2823 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:31.424217 2823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:31.424472 2823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:31.426777 2823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-1/data/instance:
uuid: "bcd6b3647e2a43b1b0e3fdd58ca84ecf"
format_stamp: "Formatted at 2026-09-29 18:32:31 on dist-test-slave-qcpv"
server_key: "c0d9cdeb297fb3ad67ebd06ab04775f5"
server_key_iv: "4ea858a635432251c912ec7271bb3b73"
server_key_version: "encryptionkey@0"
I20260929 18:32:31.427443 2823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance:
uuid: "bcd6b3647e2a43b1b0e3fdd58ca84ecf"
format_stamp: "Formatted at 2026-09-29 18:32:31 on dist-test-slave-qcpv"
server_key: "c0d9cdeb297fb3ad67ebd06ab04775f5"
server_key_iv: "4ea858a635432251c912ec7271bb3b73"
server_key_version: "encryptionkey@0"
I20260929 18:32:31.431807 2823 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:32:31.434686 2838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:31.436295 2823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:32:31.436457 2823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-1/wal
uuid: "bcd6b3647e2a43b1b0e3fdd58ca84ecf"
format_stamp: "Formatted at 2026-09-29 18:32:31 on dist-test-slave-qcpv"
server_key: "c0d9cdeb297fb3ad67ebd06ab04775f5"
server_key_iv: "4ea858a635432251c912ec7271bb3b73"
server_key_version: "encryptionkey@0"
I20260929 18:32:31.436602 2823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:31.465672 2823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:31.469182 2823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:31.469403 2823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:31.470026 2823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:32:31.470947 2823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:31.471025 2823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:31.471101 2823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:31.471153 2823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:31.482755 2823 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.66:38313
I20260929 18:32:31.482780 2951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.66:38313 every 8 connection(s)
I20260929 18:32:31.483814 2823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
I20260929 18:32:31.493125 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 2823
I20260929 18:32:31.493296 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance
I20260929 18:32:31.493695 29501 external_mini_cluster.cc:1428] Setting key eaf3e7c1035599874dc1fa409a6d5fdf
Sep 29 18:32:31 dist-test-slave-qcpv krb5kdc[2582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706751, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:32:31.503520 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:31.486538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:60107 (local address 127.28.207.126:42681)
0929 18:32:31.486717 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:31.486722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:31.487908 (+ 1186us) server_negotiation.cc:407] Connection header received
0929 18:32:31.489884 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:31.489888 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:31.489954 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:31.490045 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:31.491572 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:31.492386 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:31.494334 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:31.494814 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:31.495011 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:31.498586 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:31.498591 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:31.498595 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:31.498633 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:31.500391 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:31.501116 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:31.501122 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:31.501157 (+ 35us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:31.501261 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:31.501695 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:31.501701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:31.501706 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:31.502004 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:31.502139 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:31.503140 (+ 1001us) server_negotiation.cc:299] Negotiation successful
0929 18:32:31.503304 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
WARNING: no policy specified for kudu/127.28.207.67@KRBTEST.COM; defaulting to no policy
I20260929 18:32:31.504889 2954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:31.487002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:42681 (local address 127.28.207.66:60107)
0929 18:32:31.487685 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:31.487748 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:31.489508 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:31.490271 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:31.490280 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:31.490778 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:31.491406 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:31.491421 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:31.492633 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:31.492643 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:31.494186 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:31.494203 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:31.495097 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:31.495102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:31.495322 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:31.496279 (+ 957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:31.496298 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:31.498371 (+ 2073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:31.500530 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:31.500540 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:31.500561 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:31.500981 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:31.501402 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:31.501405 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:31.501407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:31.501583 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:31.502224 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:31.502234 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:31.502675 (+ 441us) client_negotiation.cc:769] Sending connection context
0929 18:32:31.503055 (+ 380us) client_negotiation.cc:240] Negotiation successful
0929 18:32:31.503508 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":169,"threads_started":1}
I20260929 18:32:31.507004 2952 heartbeater.cc:347] Connected to a master server at 127.28.207.126:42681
I20260929 18:32:31.507390 2952 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:31.508304 2952 heartbeater.cc:511] Master 127.28.207.126:42681 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.67@KRBTEST.COM" created.
I20260929 18:32:31.510308 2627 ts_manager.cc:194] Registered new tserver with Master: bcd6b3647e2a43b1b0e3fdd58ca84ecf (127.28.207.66:38313)
I20260929 18:32:31.511445 2627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.66@KRBTEST.COM'} at 127.28.207.66:60107
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab.
Entry for principal kudu/127.28.207.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab.
I20260929 18:32:31.532526 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.67:0
--local_ip_for_outbound_sockets=127.28.207.67
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:42681
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:44105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:31.662509 2957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:31.662832 2957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:31.662946 2957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:31.667294 2957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:31.667379 2957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:31.667443 2957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.67
I20260929 18:32:31.673818 2957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:44105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:42681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.67
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:31.675246 2957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:31.676440 2957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:32:31.688010 2957 server_base.cc:1034] running on GCE node
W20260929 18:32:31.688196 2963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:31.688086 2965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:31.688079 2962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:31.688977 2957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:31.689716 2957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:31.691030 2957 hybrid_clock.cc:648] HybridClock initialized: now 1790706751690985 us; error 68 us; skew 500 ppm
Sep 29 18:32:31 dist-test-slave-qcpv krb5kdc[2582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706751, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:31.694429 2957 init.cc:377] Logged in from keytab as kudu/127.28.207.67@KRBTEST.COM (short username kudu)
I20260929 18:32:31.695560 2957 webserver.cc:460] Webserver started at http://127.28.207.67:39811/ using document root <none> and password file <none>
I20260929 18:32:31.696182 2957 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:31.696254 2957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:31.696493 2957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:31.698623 2957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-2/data/instance:
uuid: "2589321008c14b8781938f79248a9b95"
format_stamp: "Formatted at 2026-09-29 18:32:31 on dist-test-slave-qcpv"
server_key: "0365d18d2b346b0b8364de00825823dc"
server_key_iv: "a14bba20351c6e56d0be626d94a68d2d"
server_key_version: "encryptionkey@0"
I20260929 18:32:31.699267 2957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance:
uuid: "2589321008c14b8781938f79248a9b95"
format_stamp: "Formatted at 2026-09-29 18:32:31 on dist-test-slave-qcpv"
server_key: "0365d18d2b346b0b8364de00825823dc"
server_key_iv: "a14bba20351c6e56d0be626d94a68d2d"
server_key_version: "encryptionkey@0"
I20260929 18:32:31.703529 2957 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:32:31.706708 2972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:31.708423 2957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:32:31.708616 2957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-2/wal
uuid: "2589321008c14b8781938f79248a9b95"
format_stamp: "Formatted at 2026-09-29 18:32:31 on dist-test-slave-qcpv"
server_key: "0365d18d2b346b0b8364de00825823dc"
server_key_iv: "a14bba20351c6e56d0be626d94a68d2d"
server_key_version: "encryptionkey@0"
I20260929 18:32:31.708765 2957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:31.732569 2957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:31.736346 2957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:31.736589 2957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:31.737233 2957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:32:31.738238 2957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:31.738334 2957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:31.738449 2957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:31.738497 2957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:31.750433 2957 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.67:38703
I20260929 18:32:31.750527 3085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.67:38703 every 8 connection(s)
I20260929 18:32:31.751506 2957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
I20260929 18:32:31.751895 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 2957
I20260929 18:32:31.752202 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance
I20260929 18:32:31.752668 29501 external_mini_cluster.cc:1428] Setting key 294ffba7011e4121a94ef42aa87209f6
Sep 29 18:32:31 dist-test-slave-qcpv krb5kdc[2582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706751, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:32:31.775017 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:31.753844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:40103 (local address 127.28.207.126:42681)
0929 18:32:31.754000 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:31.754005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:31.755780 (+ 1775us) server_negotiation.cc:407] Connection header received
0929 18:32:31.757758 (+ 1978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:31.757762 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:31.757867 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:31.757980 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:31.760273 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:31.761013 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:31.762979 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:31.763266 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:31.763386 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:31.767686 (+ 4300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:31.767691 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:31.767693 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:31.767729 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:31.770533 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:31.771643 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:31.771646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:31.771648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:31.771759 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:31.772376 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:31.772379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:31.772381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:31.772895 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:31.773039 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:31.774576 (+ 1537us) server_negotiation.cc:299] Negotiation successful
0929 18:32:31.774757 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 18:32:31.776916 3088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:31.754473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:42681 (local address 127.28.207.67:40103)
0929 18:32:31.755531 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:31.755623 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:31.757352 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:31.758289 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:31.758300 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:31.759201 (+ 901us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:31.759897 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:31.759915 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:31.761214 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:31.761218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:31.762701 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:31.762718 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:31.763724 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:31.763757 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:31.763989 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:31.764905 (+ 916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:31.764927 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:31.767440 (+ 2513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:31.770762 (+ 3322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:31.770771 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:31.770793 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:31.771319 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:31.771992 (+ 673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:31.771995 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:31.771997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:31.772203 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:31.773072 (+ 869us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:31.773080 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:31.773958 (+ 878us) client_negotiation.cc:769] Sending connection context
0929 18:32:31.774425 (+ 467us) client_negotiation.cc:240] Negotiation successful
0929 18:32:31.775249 (+ 824us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":120,"threads_started":1}
I20260929 18:32:31.778808 3086 heartbeater.cc:347] Connected to a master server at 127.28.207.126:42681
I20260929 18:32:31.779314 3086 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:31.780061 3086 heartbeater.cc:511] Master 127.28.207.126:42681 requested a full tablet report, sending...
I20260929 18:32:31.781388 2627 ts_manager.cc:194] Registered new tserver with Master: 2589321008c14b8781938f79248a9b95 (127.28.207.67:38703)
I20260929 18:32:31.782207 2627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.67@KRBTEST.COM'} at 127.28.207.67:40103
I20260929 18:32:31.783139 29501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/krb5kdc/test-admin.keytab.
Sep 29 18:32:31 dist-test-slave-qcpv krb5kdc[2582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:31.807144 29501 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 18:32:31.812748 3092 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:31.809747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54894 (local address 127.28.207.65:34831)
0929 18:32:31.810240 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:31.810252 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:31.810754 (+ 502us) server_negotiation.cc:407] Connection header received
0929 18:32:31.810949 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:31.810960 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:31.811464 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:31.811719 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:31.812454 (+ 735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54894: BlockingRecv error: recv got EOF from 127.0.0.1:54894 (error 108)
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":110,"threads_started":1}
W20260929 18:32:31.813799 29501 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/krb5kdc/krb5cc' of type FILE
Sep 29 18:32:31 dist-test-slave-qcpv krb5kdc[2582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:31.815168 29501 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 18:32:31 dist-test-slave-qcpv krb5kdc[2582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:31.816176 29501 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 18:32:31 dist-test-slave-qcpv krb5kdc[2582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.207.65@KRBTEST.COM
I20260929 18:32:31.828650 3092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:31.817369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54910 (local address 127.28.207.65:34831)
0929 18:32:31.817509 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:31.817520 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:31.817540 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:32:31.817761 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:31.817765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:31.817827 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:31.817948 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:31.818665 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:31.819567 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:31.820541 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:31.821056 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:31.821174 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:31.824096 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:31.824109 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:31.824125 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:31.824183 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:31.826267 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:31.826910 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:31.826917 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:31.826919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:31.827025 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:31.827434 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:31.827437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:31.827439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:31.827734 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:31.828054 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:31.828251 (+ 197us) server_negotiation.cc:299] Negotiation successful
0929 18:32:31.828461 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260929 18:32:31.833881 3092 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:31.832753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54918 (local address 127.28.207.65:34831)
0929 18:32:31.832923 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:31.832927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:31.833001 (+ 74us) server_negotiation.cc:407] Connection header received
0929 18:32:31.833145 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:31.833148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:31.833197 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:31.833294 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:31.833766 (+ 472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54918: BlockingRecv error: recv got EOF from 127.0.0.1:54918 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260929 18:32:31.834285 29501 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/krb5kdc/krb5cc)
Sep 29 18:32:31 dist-test-slave-qcpv krb5kdc[2582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:31.835721 29501 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 18:32:31 dist-test-slave-qcpv krb5kdc[2582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:31.837102 29501 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 18:32:31 dist-test-slave-qcpv krb5kdc[2582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.207.65@KRBTEST.COM
I20260929 18:32:31.848960 3092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:31.838426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54932 (local address 127.28.207.65:34831)
0929 18:32:31.838568 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:31.838571 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:31.838588 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:32:31.838705 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:31.838708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:31.838759 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:31.838903 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:31.839588 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:31.840204 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:31.841280 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:31.841569 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:31.841675 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:31.844628 (+ 2953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:31.844632 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:31.844634 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:31.844664 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:31.846526 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:31.847187 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:31.847190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:31.847192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:31.847299 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:31.847654 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:31.847657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:31.847658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:31.848015 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:31.848262 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:31.848480 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 18:32:31.848806 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260929 18:32:31.854694 3092 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:31.852796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54944 (local address 127.28.207.65:34831)
0929 18:32:31.853064 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:31.853069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:31.853405 (+ 336us) server_negotiation.cc:407] Connection header received
0929 18:32:31.853575 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:31.853578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:31.853633 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:31.853728 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:31.854520 (+ 792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54944: BlockingRecv error: recv got EOF from 127.0.0.1:54944 (error 108)
Metrics: {"server-negotiator.queue_time_us":154}
W20260929 18:32:31.855093 29501 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790706718260899-29501-0/krb5kdc/krb5cc' of type FILE
Sep 29 18:32:31 dist-test-slave-qcpv krb5kdc[2582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:31.856674 29501 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 18:32:31 dist-test-slave-qcpv krb5kdc[2582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:31.858035 29501 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 18:32:31 dist-test-slave-qcpv krb5kdc[2582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.207.65@KRBTEST.COM
I20260929 18:32:31.869485 3092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:31.859234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54950 (local address 127.28.207.65:34831)
0929 18:32:31.859463 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:31.859468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:31.859547 (+ 79us) server_negotiation.cc:407] Connection header received
0929 18:32:31.859720 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:31.859724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:31.859786 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:31.859897 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:31.860486 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:31.861104 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:31.862072 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:31.862314 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:31.862423 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:31.865238 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:31.865242 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:31.865245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:31.865275 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:31.867231 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:31.867731 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:31.867735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:31.867736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:31.867921 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:31.868275 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:31.868278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:31.868280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:31.868536 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:31.868708 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:31.869068 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 18:32:31.869268 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260929 18:32:31.870946 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 2689
I20260929 18:32:31.877243 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 2823
I20260929 18:32:31.882689 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 2957
I20260929 18:32:31.890584 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 2595
2026-09-29T18:32:31Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2498 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:32:31 dist-test-slave-qcpv krb5kdc[3106](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:32:31 dist-test-slave-qcpv krb5kdc[3106](info): set up 2 sockets
Sep 29 18:32:31 dist-test-slave-qcpv krb5kdc[3106](info): commencing operation
krb5kdc: starting...
W20260929 18:32:32.971724 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:32:33 dist-test-slave-qcpv krb5kdc[3106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:32:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:32:33Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.207.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.207.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.207.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.126.keytab.
Entry for principal oryx/127.28.207.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.126.keytab.
I20260929 18:32:33.116925 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:33133
--webserver_interface=127.28.207.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.126.keytab
--principal=oryx/127.28.207.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:42331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:33133
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:33.235540 3119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:33.235843 3119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:33.235886 3119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:33.240224 3119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:32:33.240309 3119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:33.240331 3119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:33.240375 3119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:32:33.240393 3119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:32:33.246125 3119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:42331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:33133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.126.keytab
--principal=oryx/127.28.207.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:33133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:33.247372 3119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:33.248435 3119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:33.257746 3127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:33.257719 3124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:33.257987 3125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:33.258286 3119 server_base.cc:1034] running on GCE node
I20260929 18:32:33.258939 3119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:33.260038 3119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:33.261274 3119 hybrid_clock.cc:648] HybridClock initialized: now 1790706753261228 us; error 80 us; skew 500 ppm
Sep 29 18:32:33 dist-test-slave-qcpv krb5kdc[3106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706753, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.207.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:33.264354 3119 init.cc:377] Logged in from keytab as oryx/127.28.207.126@KRBTEST.COM (short username oryx)
I20260929 18:32:33.265654 3119 webserver.cc:460] Webserver started at http://127.28.207.126:32825/ using document root <none> and password file <none>
I20260929 18:32:33.266288 3119 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:33.266366 3119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:33.266541 3119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:33.268672 3119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "1250cda9f69445ecadffa350f7d1790c"
format_stamp: "Formatted at 2026-09-29 18:32:33 on dist-test-slave-qcpv"
server_key: "4b6f844ac6d28d39376bf8655bd813cc"
server_key_iv: "84ed5e781d5c24628f75142770ae21ec"
server_key_version: "encryptionkey@0"
I20260929 18:32:33.269199 3119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "1250cda9f69445ecadffa350f7d1790c"
format_stamp: "Formatted at 2026-09-29 18:32:33 on dist-test-slave-qcpv"
server_key: "4b6f844ac6d28d39376bf8655bd813cc"
server_key_iv: "84ed5e781d5c24628f75142770ae21ec"
server_key_version: "encryptionkey@0"
I20260929 18:32:33.273330 3119 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:32:33.276149 3134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:33.277853 3119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:32:33.278023 3119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "1250cda9f69445ecadffa350f7d1790c"
format_stamp: "Formatted at 2026-09-29 18:32:33 on dist-test-slave-qcpv"
server_key: "4b6f844ac6d28d39376bf8655bd813cc"
server_key_iv: "84ed5e781d5c24628f75142770ae21ec"
server_key_version: "encryptionkey@0"
I20260929 18:32:33.278154 3119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:33.288955 3119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:33.293185 3119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:33.293428 3119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:33.301978 3119 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:33133
I20260929 18:32:33.302284 3186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:33133 every 8 connection(s)
I20260929 18:32:33.303151 3119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:32:33.306141 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 3119
I20260929 18:32:33.306305 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:32:33.306674 29501 external_mini_cluster.cc:1428] Setting key 6145ae60ecf8a7131d41d24f71f239e6
I20260929 18:32:33.307008 3187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:33.315959 3187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c: Bootstrap starting.
Sep 29 18:32:33 dist-test-slave-qcpv krb5kdc[3106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM
I20260929 18:32:33.319458 3187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:33.320582 3187 log.cc:824] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:33.322381 3190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:33.308744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33596 (local address 127.28.207.126:33133)
0929 18:32:33.309412 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:33.309430 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:33.309474 (+ 44us) server_negotiation.cc:407] Connection header received
0929 18:32:33.310212 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:33.310232 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:33.310596 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:33.311010 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:33.311870 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:33.312927 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:33.313806 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:33.314312 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:33.314500 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:33.317300 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:33.317306 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:33.317321 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:33.317353 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:33.319281 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:33.319792 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:33.319798 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:33.319803 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:33.319920 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:33.320204 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:33.320207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:33.320209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:33.320627 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:33.320841 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:33.321163 (+ 322us) server_negotiation.cc:299] Negotiation successful
0929 18:32:33.321584 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":235,"threads_started":1}
I20260929 18:32:33.323493 3187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c: No bootstrap required, opened a new log
I20260929 18:32:33.328918 3187 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } }
I20260929 18:32:33.329450 3187 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:33.329603 3187 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1250cda9f69445ecadffa350f7d1790c, State: Initialized, Role: FOLLOWER
I20260929 18:32:33.330488 3187 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } }
I20260929 18:32:33.330735 3187 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:32:33.330843 3187 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:32:33.331054 3187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:33.332419 3187 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } }
I20260929 18:32:33.332909 3187 leader_election.cc:302] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1250cda9f69445ecadffa350f7d1790c; no voters:
I20260929 18:32:33.333410 3187 leader_election.cc:288] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:32:33.333508 3192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:32:33.333746 3192 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [term 1 LEADER]: Becoming Leader. State: Replica: 1250cda9f69445ecadffa350f7d1790c, State: Running, Role: LEADER
I20260929 18:32:33.334146 3192 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } }
I20260929 18:32:33.335250 3187 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:32:33.336488 3193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1250cda9f69445ecadffa350f7d1790c. Latest consensus state: current_term: 1 leader_uuid: "1250cda9f69445ecadffa350f7d1790c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } }
I20260929 18:32:33.336712 3193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:33.337091 3194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1250cda9f69445ecadffa350f7d1790c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } }
I20260929 18:32:33.337193 3194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:33.337461 3201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:32:33.340947 3201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:32:33.348613 3201 catalog_manager.cc:1347] Generated new cluster ID: 41ad7488d0df4de6922e7f2e0903f63a
I20260929 18:32:33.348758 3201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:32:33.388440 3201 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:32:33.389766 3201 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:32:33.397632 3201 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c: Generated new TSK 0
I20260929 18:32:33.399035 3201 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.207.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.207.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.207.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.65.keytab.
Entry for principal oryx/127.28.207.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.65.keytab.
I20260929 18:32:33.452178 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.65:0
--local_ip_for_outbound_sockets=127.28.207.65
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:33133
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.65.keytab
--principal=oryx/127.28.207.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:42331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:33.589713 3213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:33.590041 3213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:33.590087 3213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:33.594477 3213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:33.594576 3213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:33.594640 3213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.65
I20260929 18:32:33.600749 3213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:42331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.65.keytab
--principal=oryx/127.28.207.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:33133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.65
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:33.601893 3213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:33.602977 3213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:33.613629 3221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:33.613933 3213 server_base.cc:1034] running on GCE node
W20260929 18:32:33.613647 3218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:33.613631 3219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:33.614636 3213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:33.615682 3213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:33.616977 3213 hybrid_clock.cc:648] HybridClock initialized: now 1790706753616921 us; error 73 us; skew 500 ppm
Sep 29 18:32:33 dist-test-slave-qcpv krb5kdc[3106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706753, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.207.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:33.620901 3213 init.cc:377] Logged in from keytab as oryx/127.28.207.65@KRBTEST.COM (short username oryx)
I20260929 18:32:33.622139 3213 webserver.cc:460] Webserver started at http://127.28.207.65:34767/ using document root <none> and password file <none>
I20260929 18:32:33.623121 3213 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:33.623239 3213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:33.623564 3213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:33.626695 3213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-0/data/instance:
uuid: "52f601bc57eb4420854cb9073fdd9caa"
format_stamp: "Formatted at 2026-09-29 18:32:33 on dist-test-slave-qcpv"
server_key: "33cc506226a82bc52db3fcd9ee5ddcf9"
server_key_iv: "dd84b2fd57b1da396f612f4d49173831"
server_key_version: "encryptionkey@0"
I20260929 18:32:33.627678 3213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance:
uuid: "52f601bc57eb4420854cb9073fdd9caa"
format_stamp: "Formatted at 2026-09-29 18:32:33 on dist-test-slave-qcpv"
server_key: "33cc506226a82bc52db3fcd9ee5ddcf9"
server_key_iv: "dd84b2fd57b1da396f612f4d49173831"
server_key_version: "encryptionkey@0"
I20260929 18:32:33.633124 3213 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 18:32:33.636850 3228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:33.638924 3213 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260929 18:32:33.639144 3213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-0/wal
uuid: "52f601bc57eb4420854cb9073fdd9caa"
format_stamp: "Formatted at 2026-09-29 18:32:33 on dist-test-slave-qcpv"
server_key: "33cc506226a82bc52db3fcd9ee5ddcf9"
server_key_iv: "dd84b2fd57b1da396f612f4d49173831"
server_key_version: "encryptionkey@0"
I20260929 18:32:33.639313 3213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:33.654120 3213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:33.659415 3213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:33.659638 3213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:33.660368 3213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:32:33.661444 3213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:33.661526 3213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:33.661607 3213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:33.661659 3213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:33.676370 3213 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.65:37323
I20260929 18:32:33.676398 3341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.65:37323 every 8 connection(s)
I20260929 18:32:33.677933 3213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
I20260929 18:32:33.683224 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 3213
I20260929 18:32:33.683387 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance
I20260929 18:32:33.683732 29501 external_mini_cluster.cc:1428] Setting key 19e67a480c8201ef0799d6f3c477f6d3
Sep 29 18:32:33 dist-test-slave-qcpv krb5kdc[3106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706753, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.207.65@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.28.207.66@KRBTEST.COM; defaulting to no policy
I20260929 18:32:33.697897 3190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:33.680592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:48489 (local address 127.28.207.126:33133)
0929 18:32:33.680749 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:33.680753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:33.682031 (+ 1278us) server_negotiation.cc:407] Connection header received
0929 18:32:33.684218 (+ 2187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:33.684223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:33.684322 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:33.684473 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:33.686110 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:33.686945 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:33.688484 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:33.688980 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:33.689137 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:33.692848 (+ 3711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:33.692853 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:33.692855 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:33.692890 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:33.694684 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:33.695628 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:33.695631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:33.695632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:33.695728 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:33.696335 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:33.696338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:33.696339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:33.696575 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:33.696668 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:33.697462 (+ 794us) server_negotiation.cc:299] Negotiation successful
0929 18:32:33.697636 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 18:32:33.699250 3344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:33.681052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:33133 (local address 127.28.207.65:48489)
0929 18:32:33.681785 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:33.681860 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:33.683813 (+ 1953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:33.684733 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:33.684751 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:33.685195 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:33.685887 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:33.685906 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:33.687115 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:33.687122 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:33.688329 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:33.688340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:33.689181 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:33.689184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:33.689348 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:33.690149 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:33.690167 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:33.692516 (+ 2349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:33.694893 (+ 2377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:33.694904 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:33.694933 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:33.695448 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:33.695869 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:33.695875 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:33.695878 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:33.696104 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:33.696732 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:33.696746 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:33.697149 (+ 403us) client_negotiation.cc:769] Sending connection context
0929 18:32:33.697341 (+ 192us) client_negotiation.cc:240] Negotiation successful
0929 18:32:33.697758 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":172,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.207.66@KRBTEST.COM" created.
I20260929 18:32:33.701107 3342 heartbeater.cc:347] Connected to a master server at 127.28.207.126:33133
I20260929 18:32:33.701488 3342 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:33.702337 3342 heartbeater.cc:511] Master 127.28.207.126:33133 requested a full tablet report, sending...
I20260929 18:32:33.704495 3151 ts_manager.cc:194] Registered new tserver with Master: 52f601bc57eb4420854cb9073fdd9caa (127.28.207.65:37323)
I20260929 18:32:33.706168 3151 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.207.65@KRBTEST.COM'} at 127.28.207.65:48489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.207.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.66.keytab.
Entry for principal oryx/127.28.207.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.66.keytab.
I20260929 18:32:33.719321 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.66:0
--local_ip_for_outbound_sockets=127.28.207.66
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:33133
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.66.keytab
--principal=oryx/127.28.207.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:42331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:33.836529 3347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:33.836829 3347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:33.836876 3347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:33.841461 3347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:33.841563 3347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:33.841629 3347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.66
I20260929 18:32:33.848181 3347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:42331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.66.keytab
--principal=oryx/127.28.207.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:33133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.66
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:33.849633 3347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:33.851097 3347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:33.862351 3352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:33.862351 3353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:33.862344 3355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:33.862732 3347 server_base.cc:1034] running on GCE node
I20260929 18:32:33.863529 3347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:33.864477 3347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:33.865700 3347 hybrid_clock.cc:648] HybridClock initialized: now 1790706753865673 us; error 43 us; skew 500 ppm
Sep 29 18:32:33 dist-test-slave-qcpv krb5kdc[3106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706753, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.207.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:33.869586 3347 init.cc:377] Logged in from keytab as oryx/127.28.207.66@KRBTEST.COM (short username oryx)
I20260929 18:32:33.871021 3347 webserver.cc:460] Webserver started at http://127.28.207.66:39381/ using document root <none> and password file <none>
I20260929 18:32:33.871767 3347 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:33.871869 3347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:33.872151 3347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:33.874984 3347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-1/data/instance:
uuid: "af2018baa42e42b9b287a44255461376"
format_stamp: "Formatted at 2026-09-29 18:32:33 on dist-test-slave-qcpv"
server_key: "202346058d07fcf6406cb7b3d08a752c"
server_key_iv: "c250200120a9c76f70f01aef047875c1"
server_key_version: "encryptionkey@0"
I20260929 18:32:33.876746 3347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance:
uuid: "af2018baa42e42b9b287a44255461376"
format_stamp: "Formatted at 2026-09-29 18:32:33 on dist-test-slave-qcpv"
server_key: "202346058d07fcf6406cb7b3d08a752c"
server_key_iv: "c250200120a9c76f70f01aef047875c1"
server_key_version: "encryptionkey@0"
I20260929 18:32:33.882397 3347 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260929 18:32:33.886075 3362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:33.888121 3347 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 18:32:33.888335 3347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-1/wal
uuid: "af2018baa42e42b9b287a44255461376"
format_stamp: "Formatted at 2026-09-29 18:32:33 on dist-test-slave-qcpv"
server_key: "202346058d07fcf6406cb7b3d08a752c"
server_key_iv: "c250200120a9c76f70f01aef047875c1"
server_key_version: "encryptionkey@0"
I20260929 18:32:33.888500 3347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:33.915755 3347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:33.920128 3347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:33.920382 3347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:33.921155 3347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:32:33.922224 3347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:33.922302 3347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:33.922385 3347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:33.922430 3347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:33.935745 3347 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.66:44677
I20260929 18:32:33.935757 3475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.66:44677 every 8 connection(s)
I20260929 18:32:33.937129 3347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
I20260929 18:32:33.938378 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 3347
I20260929 18:32:33.938519 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance
I20260929 18:32:33.938807 29501 external_mini_cluster.cc:1428] Setting key 0a096c2fa72dd6dc6a469d99faa05f06
WARNING: no policy specified for oryx/127.28.207.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.207.67@KRBTEST.COM" created.
Sep 29 18:32:33 dist-test-slave-qcpv krb5kdc[3106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706753, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.207.66@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM
I20260929 18:32:33.959995 3190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:33.939656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:50657 (local address 127.28.207.126:33133)
0929 18:32:33.939815 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:33.939820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:33.941074 (+ 1254us) server_negotiation.cc:407] Connection header received
0929 18:32:33.942850 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:33.942886 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:33.942963 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:33.943126 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:33.944678 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:33.945672 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:33.948391 (+ 2719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:33.948823 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:33.948980 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:33.954060 (+ 5080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:33.954069 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:33.954076 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:33.954130 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:33.956070 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:33.956949 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:33.956954 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:33.956955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:33.957078 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:33.957623 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:33.957627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:33.957628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:33.958105 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:33.958267 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:33.959495 (+ 1228us) server_negotiation.cc:299] Negotiation successful
0929 18:32:33.959695 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 18:32:33.960602 3478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:33.940253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:33133 (local address 127.28.207.66:50657)
0929 18:32:33.940867 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:33.940929 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:33.942566 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:33.943375 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:33.943384 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:33.943859 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:33.944480 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:33.944495 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:33.945928 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:33.945934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:33.948197 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:33.948211 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:33.949306 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:33.949309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:33.949502 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:33.950339 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:33.950359 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:33.953746 (+ 3387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:33.956292 (+ 2546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:33.956303 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:33.956327 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:33.956775 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:33.957216 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:33.957222 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:33.957227 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:33.957443 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:33.958280 (+ 837us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:33.958291 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:33.958720 (+ 429us) client_negotiation.cc:769] Sending connection context
0929 18:32:33.959197 (+ 477us) client_negotiation.cc:240] Negotiation successful
0929 18:32:33.959559 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":156,"threads_started":1}
I20260929 18:32:33.962827 3476 heartbeater.cc:347] Connected to a master server at 127.28.207.126:33133
I20260929 18:32:33.963260 3476 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:33.964008 3476 heartbeater.cc:511] Master 127.28.207.126:33133 requested a full tablet report, sending...
I20260929 18:32:33.965397 3151 ts_manager.cc:194] Registered new tserver with Master: af2018baa42e42b9b287a44255461376 (127.28.207.66:44677)
I20260929 18:32:33.966073 3151 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.207.66@KRBTEST.COM'} at 127.28.207.66:50657
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.207.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.67.keytab.
Entry for principal oryx/127.28.207.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.67.keytab.
I20260929 18:32:33.976827 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.67:0
--local_ip_for_outbound_sockets=127.28.207.67
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:33133
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.67.keytab
--principal=oryx/127.28.207.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:42331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:34.102373 3481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:34.102684 3481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:34.102738 3481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:34.107245 3481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:34.107333 3481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:34.107398 3481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.67
I20260929 18:32:34.112979 3481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:42331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.67.keytab
--principal=oryx/127.28.207.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:33133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.67
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:34.114096 3481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:34.115151 3481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:34.125663 3487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:34.125938 3486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:34.126060 3481 server_base.cc:1034] running on GCE node
W20260929 18:32:34.125718 3489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:34.126570 3481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:34.127404 3481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:34.128592 3481 hybrid_clock.cc:648] HybridClock initialized: now 1790706754128579 us; error 37 us; skew 500 ppm
Sep 29 18:32:34 dist-test-slave-qcpv krb5kdc[3106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706754, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.207.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:34.132156 3481 init.cc:377] Logged in from keytab as oryx/127.28.207.67@KRBTEST.COM (short username oryx)
I20260929 18:32:34.133432 3481 webserver.cc:460] Webserver started at http://127.28.207.67:35729/ using document root <none> and password file <none>
I20260929 18:32:34.134224 3481 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:34.134341 3481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:34.134631 3481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:34.137894 3481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-2/data/instance:
uuid: "37153dd37c404c03a93687dad1b4cdca"
format_stamp: "Formatted at 2026-09-29 18:32:34 on dist-test-slave-qcpv"
server_key: "20eac7ce9446e3012cab8ef9941bde39"
server_key_iv: "e77820779dedf9999f590b80ea482052"
server_key_version: "encryptionkey@0"
I20260929 18:32:34.138619 3481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance:
uuid: "37153dd37c404c03a93687dad1b4cdca"
format_stamp: "Formatted at 2026-09-29 18:32:34 on dist-test-slave-qcpv"
server_key: "20eac7ce9446e3012cab8ef9941bde39"
server_key_iv: "e77820779dedf9999f590b80ea482052"
server_key_version: "encryptionkey@0"
I20260929 18:32:34.144605 3481 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.001s
I20260929 18:32:34.147810 3496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:34.149611 3481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 18:32:34.149816 3481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-2/wal
uuid: "37153dd37c404c03a93687dad1b4cdca"
format_stamp: "Formatted at 2026-09-29 18:32:34 on dist-test-slave-qcpv"
server_key: "20eac7ce9446e3012cab8ef9941bde39"
server_key_iv: "e77820779dedf9999f590b80ea482052"
server_key_version: "encryptionkey@0"
I20260929 18:32:34.149984 3481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:34.165890 3481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:34.169626 3481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:34.169884 3481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:34.170542 3481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:32:34.171495 3481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:34.171568 3481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:34.171638 3481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:34.171674 3481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:34.183321 3481 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.67:36657
I20260929 18:32:34.183382 3609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.67:36657 every 8 connection(s)
I20260929 18:32:34.184415 3481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
I20260929 18:32:34.185837 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 3481
I20260929 18:32:34.185995 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance
I20260929 18:32:34.186267 29501 external_mini_cluster.cc:1428] Setting key 0ac0ede4be6cc92b0681a4d3be31f413
Sep 29 18:32:34 dist-test-slave-qcpv krb5kdc[3106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706754, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.207.67@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM
I20260929 18:32:34.202976 3190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:34.187051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:55119 (local address 127.28.207.126:33133)
0929 18:32:34.187269 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:34.187275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:34.188827 (+ 1552us) server_negotiation.cc:407] Connection header received
0929 18:32:34.190468 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:34.190471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:34.190535 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:34.190631 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:34.192401 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.193012 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.194384 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.194677 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.194816 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:34.198092 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:34.198096 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:34.198098 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:34.198130 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:34.200017 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:34.200748 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:34.200753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:34.200755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:34.200932 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:34.201455 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:34.201458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:34.201460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:34.201779 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:34.201988 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:34.202593 (+ 605us) server_negotiation.cc:299] Negotiation successful
0929 18:32:34.202742 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 18:32:34.203958 3612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:34.187724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:33133 (local address 127.28.207.67:55119)
0929 18:32:34.188584 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:34.188679 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:34.190193 (+ 1514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:34.190967 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:34.190978 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:34.191513 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:34.192206 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:34.192223 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.193157 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.193160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:34.194154 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:34.194164 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.194936 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.194941 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:34.195103 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:34.196027 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:34.196045 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:34.197847 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:34.200129 (+ 2282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:34.200137 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:34.200154 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:34.200563 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:34.201056 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:34.201059 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:34.201060 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:34.201321 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:34.201953 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:34.201960 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:34.202255 (+ 295us) client_negotiation.cc:769] Sending connection context
0929 18:32:34.202445 (+ 190us) client_negotiation.cc:240] Negotiation successful
0929 18:32:34.202747 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":197,"threads_started":1}
I20260929 18:32:34.205430 3610 heartbeater.cc:347] Connected to a master server at 127.28.207.126:33133
I20260929 18:32:34.205825 3610 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:34.206534 3610 heartbeater.cc:511] Master 127.28.207.126:33133 requested a full tablet report, sending...
I20260929 18:32:34.208067 3151 ts_manager.cc:194] Registered new tserver with Master: 37153dd37c404c03a93687dad1b4cdca (127.28.207.67:36657)
I20260929 18:32:34.208752 3151 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.207.67@KRBTEST.COM'} at 127.28.207.67:55119
I20260929 18:32:34.214249 29501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.207.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.207.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.207.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.125.keytab.
Entry for principal oryx/127.28.207.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.125.keytab.
I20260929 18:32:34.247819 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.125:32877
--webserver_interface=127.28.207.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.125.keytab
--principal=oryx/127.28.207.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:42331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:33133,127.28.207.125:32877
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:34.366556 3615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:34.366940 3615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:34.367043 3615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:34.371284 3615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:32:34.371414 3615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:34.371452 3615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:34.371474 3615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:32:34.371490 3615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:32:34.377210 3615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:42331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:33133,127.28.207.125:32877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.125.keytab
--principal=oryx/127.28.207.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.125:32877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.207.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
W20260929 18:32:34.377444 3615 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.28.207.126:33133,127.28.207.125:32877'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 18:32:34.378434 3615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:34.379810 3615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:34.389700 3620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:34.389820 3621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:34.389701 3623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:34.390487 3615 server_base.cc:1034] running on GCE node
I20260929 18:32:34.391232 3615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:34.392625 3615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:34.393929 3615 hybrid_clock.cc:648] HybridClock initialized: now 1790706754393914 us; error 48 us; skew 500 ppm
Sep 29 18:32:34 dist-test-slave-qcpv krb5kdc[3106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706754, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.207.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:34.397606 3615 init.cc:377] Logged in from keytab as oryx/127.28.207.125@KRBTEST.COM (short username oryx)
I20260929 18:32:34.398676 3615 webserver.cc:460] Webserver started at http://127.28.207.125:43233/ using document root <none> and password file <none>
I20260929 18:32:34.399369 3615 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:34.399466 3615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:34.399679 3615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:34.401909 3615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-1/data/instance:
uuid: "9f296fd5b96e4975b59022170377b843"
format_stamp: "Formatted at 2026-09-29 18:32:34 on dist-test-slave-qcpv"
server_key: "47c0894032738405b170fdbd80771a01"
server_key_iv: "314e3c411f229f79e648f3bbbb18d785"
server_key_version: "encryptionkey@0"
I20260929 18:32:34.402503 3615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-1/wal/instance:
uuid: "9f296fd5b96e4975b59022170377b843"
format_stamp: "Formatted at 2026-09-29 18:32:34 on dist-test-slave-qcpv"
server_key: "47c0894032738405b170fdbd80771a01"
server_key_iv: "314e3c411f229f79e648f3bbbb18d785"
server_key_version: "encryptionkey@0"
I20260929 18:32:34.406710 3615 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:32:34.409534 3630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:34.411150 3615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:32:34.411316 3615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-1/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-1/wal
uuid: "9f296fd5b96e4975b59022170377b843"
format_stamp: "Formatted at 2026-09-29 18:32:34 on dist-test-slave-qcpv"
server_key: "47c0894032738405b170fdbd80771a01"
server_key_iv: "314e3c411f229f79e648f3bbbb18d785"
server_key_version: "encryptionkey@0"
I20260929 18:32:34.411440 3615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:34.424428 3615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:34.428125 3615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:34.428351 3615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:34.436614 3615 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.125:32877
I20260929 18:32:34.436666 3682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.125:32877 every 8 connection(s)
I20260929 18:32:34.437727 3615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-1/data/info.pb
I20260929 18:32:34.441754 3683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:34.446120 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 3615
I20260929 18:32:34.447431 3683 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:32:34 dist-test-slave-qcpv krb5kdc[3106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.207.125@KRBTEST.COM
Sep 29 18:32:34 dist-test-slave-qcpv krb5kdc[3106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706754, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.207.125@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM
I20260929 18:32:34.462647 3190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:34.449764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33606 (local address 127.28.207.126:33133)
0929 18:32:34.449953 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:34.449958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:34.450255 (+ 297us) server_negotiation.cc:407] Connection header received
0929 18:32:34.450634 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:34.450638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:34.450713 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:34.450812 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:34.452096 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.452678 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.454087 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.454408 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.454524 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:34.458073 (+ 3549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:34.458076 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:34.458078 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:34.458104 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:34.459911 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:34.460519 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:34.460522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:34.460524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:34.460613 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:34.461094 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:34.461097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:34.461099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:34.461440 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:34.461555 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:34.462227 (+ 672us) server_negotiation.cc:299] Negotiation successful
0929 18:32:34.462430 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 18:32:34.463665 3687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:34.449607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:33133 (local address 127.0.0.1:33606)
0929 18:32:34.450079 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:34.450131 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:34.450400 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:34.450918 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:34.450925 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:34.451562 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:34.451894 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:34.451907 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.452829 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.452833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:34.453871 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:34.453881 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.454574 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.454578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:34.454798 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:34.455629 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:34.455647 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:34.457837 (+ 2190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:34.460076 (+ 2239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:34.460089 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:34.460101 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:34.460385 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:34.460728 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:34.460731 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:34.460734 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:34.460971 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:34.461562 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:34.461572 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:34.461898 (+ 326us) client_negotiation.cc:769] Sending connection context
0929 18:32:34.462114 (+ 216us) client_negotiation.cc:240] Negotiation successful
0929 18:32:34.462529 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":39,"spinlock_wait_cycles":14720,"thread_start_us":107,"threads_started":1}
I20260929 18:32:34.463675 3686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:34.448127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50658 (local address 127.28.207.125:32877)
0929 18:32:34.448751 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:34.448773 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:34.448826 (+ 53us) server_negotiation.cc:407] Connection header received
0929 18:32:34.449910 (+ 1084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:34.449938 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:34.450398 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:34.450681 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:34.451630 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.452708 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.453586 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.454164 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.454337 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:34.457382 (+ 3045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:34.457389 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:34.457403 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:34.457438 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:34.460170 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:34.460717 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:34.460725 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:34.460727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:34.460971 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:34.461371 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:34.461376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:34.461378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:34.461891 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:34.462106 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:34.462387 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 18:32:34.462785 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":259,"server-negotiator.queue_time_us":465,"spinlock_wait_cycles":4864,"thread_start_us":207,"threads_started":1}
I20260929 18:32:34.465628 3683 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.207.125" port: 32877 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:32:34 dist-test-slave-qcpv krb5kdc[3106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706754, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.207.125@KRBTEST.COM for oryx/127.28.207.125@KRBTEST.COM
I20260929 18:32:34.476503 3687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:34.466290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.125:32877 (local address 127.0.0.1:50664)
0929 18:32:34.466523 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:34.466537 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:34.466624 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:34.466837 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:34.466840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:34.467063 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:34.467173 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:34.467178 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.467913 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.467915 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:34.468968 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:34.468976 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.469635 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.469638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:34.469725 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:34.470233 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:34.470245 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:34.472033 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:34.474196 (+ 2163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:34.474198 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:34.474200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:34.474412 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:34.474819 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:34.474822 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:34.474823 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:34.474979 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:34.476076 (+ 1097us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:34.476078 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:34.476175 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 18:32:34.476262 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 18:32:34.476374 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260929 18:32:34.476962 3686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:34.466324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50664 (local address 127.28.207.125:32877)
0929 18:32:34.466426 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:34.466430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:34.466638 (+ 208us) server_negotiation.cc:407] Connection header received
0929 18:32:34.466686 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:34.466688 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:34.466727 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:34.466815 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:34.467302 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.467819 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.469159 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.469470 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.469595 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:34.472334 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:34.472344 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:34.472347 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:34.472376 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:34.474054 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:34.474597 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:34.474601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:34.474602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:34.474700 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:34.475637 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:34.475641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:34.475642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:34.475860 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:34.475960 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:34.476519 (+ 559us) server_negotiation.cc:299] Negotiation successful
0929 18:32:34.476723 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 18:32:34.481043 3683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843: Bootstrap starting.
I20260929 18:32:34.484166 3683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:34.485095 3683 log.cc:824] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:34.487852 3683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843: No bootstrap required, opened a new log
I20260929 18:32:34.491511 3683 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 32877 } }
I20260929 18:32:34.491786 3683 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:34.491822 3683 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f296fd5b96e4975b59022170377b843, State: Initialized, Role: FOLLOWER
I20260929 18:32:34.492383 3683 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 32877 } }
I20260929 18:32:34.493114 3689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 32877 } } }
I20260929 18:32:34.493360 3689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:32:34.493960 3683 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 18:32:34.499775 3700 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:32:34.499912 3700 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:32:34.503299 3615 master_runner.cc:428] Detected that this master 9f296fd5b96e4975b59022170377b843 is joining an existing cluster
I20260929 18:32:34.503386 3615 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.207.125:32877
I20260929 18:32:34.515872 3686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:34.506014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50666 (local address 127.28.207.125:32877)
0929 18:32:34.506155 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:34.506161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:34.506622 (+ 461us) server_negotiation.cc:407] Connection header received
0929 18:32:34.506738 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:34.506741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:34.506783 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:34.506833 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:34.507405 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.507926 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.508939 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.509474 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.509600 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:34.511961 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:34.511966 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:34.511969 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:34.511993 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:34.513852 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:34.514463 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:34.514470 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:34.514472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:34.514560 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:34.514843 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:34.514846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:34.514848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:34.515065 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:34.515195 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:34.515489 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 18:32:34.515642 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":43}
I20260929 18:32:34.515892 3707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:34.506174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.125:32877 (local address 127.0.0.1:50666)
0929 18:32:34.506515 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:34.506535 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:34.506650 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:34.506994 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:34.506996 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:34.507176 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:34.507318 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:34.507324 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.508031 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.508038 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:34.508788 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:34.508796 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.509772 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.509777 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:34.509894 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:34.511207 (+ 1313us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:34.511224 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:34.511787 (+ 563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:34.513991 (+ 2204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:34.513993 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:34.513995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:34.514358 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:34.514656 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:34.514659 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:34.514661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:34.514750 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:34.515184 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:34.515187 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:34.515309 (+ 122us) client_negotiation.cc:769] Sending connection context
0929 18:32:34.515443 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 18:32:34.515724 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":104,"thread_start_us":124,"threads_started":1}
I20260929 18:32:34.516947 3706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:34.505867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:33133 (local address 127.0.0.1:33608)
0929 18:32:34.506250 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:34.506264 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:34.506399 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:34.506741 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:34.506743 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:34.507033 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:34.507184 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:34.507192 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.508568 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.508571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:34.509319 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:34.509328 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.509832 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.509835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:34.509945 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:34.510655 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:34.510667 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:34.511093 (+ 426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:34.514027 (+ 2934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:34.514031 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:34.514034 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:34.514394 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:34.514972 (+ 578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:34.514975 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:34.514977 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:34.515074 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:34.516207 (+ 1133us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:34.516210 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:34.516394 (+ 184us) client_negotiation.cc:769] Sending connection context
0929 18:32:34.516536 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 18:32:34.516714 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":32,"thread_start_us":113,"threads_started":1}
I20260929 18:32:34.516950 3190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:34.505920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33608 (local address 127.28.207.126:33133)
0929 18:32:34.506065 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:34.506070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:34.506345 (+ 275us) server_negotiation.cc:407] Connection header received
0929 18:32:34.506535 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:34.506539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:34.506614 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:34.506719 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:34.507443 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.508427 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.509451 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.509709 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.509829 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:34.511253 (+ 1424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:34.511257 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:34.511260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:34.511291 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:34.512847 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:34.514657 (+ 1810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:34.514661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:34.514663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:34.514799 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:34.515501 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:34.515517 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:34.515519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:34.515806 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:34.515891 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:34.516645 (+ 754us) server_negotiation.cc:299] Negotiation successful
0929 18:32:34.516795 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
WARNING: no policy specified for oryx/127.28.207.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.207.124@KRBTEST.COM" created.
Sep 29 18:32:34 dist-test-slave-qcpv krb5kdc[3106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706753, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.207.126@KRBTEST.COM for oryx/127.28.207.125@KRBTEST.COM
I20260929 18:32:34.536597 3686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:34.521979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50670 (local address 127.28.207.125:32877)
0929 18:32:34.522235 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:34.522241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:34.522924 (+ 683us) server_negotiation.cc:407] Connection header received
0929 18:32:34.523348 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:34.523355 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:34.523431 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:34.523540 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:34.524631 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.525471 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.527221 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.527614 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.527730 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:34.531833 (+ 4103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:34.531838 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:34.531845 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:34.531882 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:34.533981 (+ 2099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:34.534761 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:34.534767 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:34.534769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:34.534913 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:34.535269 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:34.535276 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:34.535280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:34.535576 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:34.535699 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:34.536227 (+ 528us) server_negotiation.cc:299] Negotiation successful
0929 18:32:34.536385 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:32:34.536960 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:34.521940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.125:32877 (local address 127.0.0.1:50670)
0929 18:32:34.522665 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:34.522757 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:34.523047 (+ 290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:34.523620 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:34.523629 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:34.524277 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:34.524469 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:34.524480 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.525615 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.525619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:34.526846 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:34.527049 (+ 203us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.527765 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.527769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:34.527926 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:34.528733 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:34.528748 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:34.531605 (+ 2857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:34.534188 (+ 2583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:34.534202 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:34.534211 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:34.534608 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:34.535027 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:34.535031 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:34.535033 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:34.535137 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:34.535699 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:34.535709 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:34.535896 (+ 187us) client_negotiation.cc:769] Sending connection context
0929 18:32:34.536078 (+ 182us) client_negotiation.cc:240] Negotiation successful
0929 18:32:34.536549 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":107,"threads_started":1}
I20260929 18:32:34.539021 3151 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.207.125:32877: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9f296fd5b96e4975b59022170377b843"
member_type: NON_VOTER
last_known_addr {
host: "127.28.207.125"
port: 32877
}
attrs {
promote: true
}
}
dest_uuid: "1250cda9f69445ecadffa350f7d1790c"
cas_config_opid_index: -1
I20260929 18:32:34.539732 3151 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: NON_VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: true } }
I20260929 18:32:34.541817 3713 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 9f296fd5b96e4975b59022170377b843 (127.28.207.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: NON_VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: true } } }
I20260929 18:32:34.542783 3657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:34.544320 3657 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [term 1 FOLLOWER]: Refusing update from remote peer 1250cda9f69445ecadffa350f7d1790c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 18:32:34.545193 3712 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: NON_VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:32:34.545433 3713 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.207.125:32877
I20260929 18:32:34.545974 3713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1250cda9f69445ecadffa350f7d1790c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: NON_VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: true } } }
I20260929 18:32:34.546095 3615 master.cc:537] Master@127.28.207.125:32877 shutting down...
I20260929 18:32:34.546095 3713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.207.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.124.keytab.
Entry for principal oryx/127.28.207.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.124.keytab.
W20260929 18:32:34.550436 3136 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260929 18:32:34.551218 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.124:45571
--webserver_interface=127.28.207.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.124.keytab
--principal=oryx/127.28.207.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:42331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:33133,127.28.207.125:32877,127.28.207.124:45571
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:34.551999 3136 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c -> Peer 9f296fd5b96e4975b59022170377b843 (127.28.207.125:32877): Couldn't send request to peer 9f296fd5b96e4975b59022170377b843. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 18:32:34.554278 3615 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:32:34.554634 3615 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:34.554790 3615 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843: stopping tablet replica
I20260929 18:32:34.561306 3615 master.cc:552] Master@127.28.207.125:32877 shutdown complete.
I20260929 18:32:34.561436 3615 master_runner.cc:305] Clearing existing system tablet
I20260929 18:32:34.561779 3615 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:34.561885 3615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:34.563915 3615 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 18:32:34.565091 3720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:34.565448 3615 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:34.565742 3615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:32:34.565824 3615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-1/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-1/wal
uuid: "9f296fd5b96e4975b59022170377b843"
format_stamp: "Formatted at 2026-09-29 18:32:34 on dist-test-slave-qcpv"
server_key: "47c0894032738405b170fdbd80771a01"
server_key_iv: "314e3c411f229f79e648f3bbbb18d785"
server_key_version: "encryptionkey@0"
I20260929 18:32:34.568192 3615 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:32:34.570726 3615 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 18:32:34.570932 3615 log.cc:1196] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 18:32:34.571302 3615 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843: Deleting consensus metadata
I20260929 18:32:34.571524 3615 master_runner.cc:315] Copying system tablet from 127.28.207.126:33133
I20260929 18:32:34.572968 3615 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843: tablet copy: Beginning tablet copy session from remote peer at address 127.28.207.126:33133
I20260929 18:32:34.583762 3726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:34.573815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:33133 (local address 127.0.0.1:33624)
0929 18:32:34.574179 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:34.574195 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:34.574340 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:34.574742 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:34.574745 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:34.575048 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:34.575200 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:34.575207 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.576466 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.576471 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:34.577476 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:34.577490 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.578161 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.578167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:34.578275 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:34.578968 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:34.578983 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:34.579467 (+ 484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:34.581754 (+ 2287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:34.581758 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:34.581760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:34.582124 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:34.582464 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:34.582467 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:34.582469 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:34.582565 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:34.583117 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:34.583120 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:34.583260 (+ 140us) client_negotiation.cc:769] Sending connection context
0929 18:32:34.583381 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 18:32:34.583539 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":99,"threads_started":1}
I20260929 18:32:34.583812 3190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:34.573869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33624 (local address 127.28.207.126:33133)
0929 18:32:34.574068 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:34.574073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:34.574309 (+ 236us) server_negotiation.cc:407] Connection header received
0929 18:32:34.574468 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:34.574472 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:34.574544 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:34.574684 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:34.575394 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.576317 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.577630 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.578016 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.578129 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:34.579640 (+ 1511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:34.579644 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:34.579648 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:34.579678 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:34.581613 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:34.582246 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:34.582250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:34.582252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:34.582354 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:34.582673 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:34.582676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:34.582678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:34.583000 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:34.583103 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:34.583468 (+ 365us) server_negotiation.cc:299] Negotiation successful
0929 18:32:34.583615 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 18:32:34.584889 3171 tablet_copy_service.cc:140] P 1250cda9f69445ecadffa350f7d1790c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9f296fd5b96e4975b59022170377b843 ({username='oryx', principal='oryx/127.28.207.125@KRBTEST.COM'} at 127.0.0.1:33624)
I20260929 18:32:34.585093 3171 tablet_copy_service.cc:161] P 1250cda9f69445ecadffa350f7d1790c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9f296fd5b96e4975b59022170377b843 at {username='oryx', principal='oryx/127.28.207.125@KRBTEST.COM'} at 127.0.0.1:33624: session id = 9f296fd5b96e4975b59022170377b843-00000000000000000000000000000000
I20260929 18:32:34.587046 3171 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 18:32:34.588876 3615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 18:32:34.591728 3615 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843: tablet copy: Starting download of 0 data blocks...
I20260929 18:32:34.591954 3615 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843: tablet copy: Starting download of 1 WAL segments...
I20260929 18:32:34.594100 3615 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 18:32:34.596581 3171 tablet_copy_service.cc:342] P 1250cda9f69445ecadffa350f7d1790c: Request end of tablet copy session 9f296fd5b96e4975b59022170377b843-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.207.125@KRBTEST.COM'} at 127.0.0.1:33624
I20260929 18:32:34.596740 3171 tablet_copy_service.cc:433] P 1250cda9f69445ecadffa350f7d1790c: ending tablet copy session 9f296fd5b96e4975b59022170377b843-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9f296fd5b96e4975b59022170377b843
I20260929 18:32:34.605979 3615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:34.608422 3727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:34.608536 3728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:34.612253 3730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:34.612463 3615 server_base.cc:1034] running on GCE node
I20260929 18:32:34.612782 3615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:34.613149 3615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:34.614297 3615 hybrid_clock.cc:648] HybridClock initialized: now 1790706754614267 us; error 54 us; skew 500 ppm
Sep 29 18:32:34 dist-test-slave-qcpv krb5kdc[3106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706754, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.207.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:34.616166 3615 init.cc:377] Logged in from keytab as oryx/127.28.207.125@KRBTEST.COM (short username oryx)
I20260929 18:32:34.616772 3615 webserver.cc:460] Webserver started at http://127.28.207.125:35263/ using document root <none> and password file <none>
I20260929 18:32:34.616991 3615 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:34.617038 3615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:34.618644 3615 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 18:32:34.619920 3737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:34.620292 3615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:32:34.620380 3615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-1/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-1/wal
uuid: "9f296fd5b96e4975b59022170377b843"
format_stamp: "Formatted at 2026-09-29 18:32:34 on dist-test-slave-qcpv"
server_key: "47c0894032738405b170fdbd80771a01"
server_key_iv: "314e3c411f229f79e648f3bbbb18d785"
server_key_version: "encryptionkey@0"
I20260929 18:32:34.620479 3615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:34.645848 3615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:34.648550 3615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:34.654690 3615 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.125:32877
I20260929 18:32:34.654907 3789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.125:32877 every 8 connection(s)
I20260929 18:32:34.655525 3615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-1/data/info.pb
I20260929 18:32:34.658366 3790 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:32:34.659751 3790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843: Bootstrap starting.
I20260929 18:32:34.671634 3790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:32:34.672348 3790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843: Bootstrap complete.
I20260929 18:32:34.673238 3790 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: NON_VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: true } }
I20260929 18:32:34.673496 3790 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9f296fd5b96e4975b59022170377b843, State: Initialized, Role: LEARNER
I20260929 18:32:34.673676 3790 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: NON_VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: true } }
I20260929 18:32:34.674304 3790 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:32:34.674295 3793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: NON_VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: true } } }
I20260929 18:32:34.674543 3793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [sys.catalog]: This master's current role is: LEARNER
I20260929 18:32:34.677763 3804 catalog_manager.cc:1259] Loaded cluster ID: 41ad7488d0df4de6922e7f2e0903f63a
I20260929 18:32:34.679980 3804 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843: loading cluster ID for follower catalog manager: success
I20260929 18:32:34.682606 3804 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843: acquiring CA information for follower catalog manager: success
I20260929 18:32:34.684423 3804 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 18:32:34.700376 3718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:34.700716 3718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:34.700809 3718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:34.705334 3718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:32:34.705437 3718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:34.705474 3718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:34.705533 3718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:32:34.705590 3718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:32:34.711282 3342 heartbeater.cc:503] Master 127.28.207.126:33133 was elected leader, sending a full tablet report...
I20260929 18:32:34.711861 3718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:42331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:33133,127.28.207.125:32877,127.28.207.124:45571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.124.keytab
--principal=oryx/127.28.207.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.124:45571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.207.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:34.713505 3718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:34.714614 3718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:34.724805 3812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:34.724805 3810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:34.724987 3809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:34.725644 3718 server_base.cc:1034] running on GCE node
I20260929 18:32:34.726229 3718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:34.727458 3718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:34.728644 3718 hybrid_clock.cc:648] HybridClock initialized: now 1790706754728630 us; error 49 us; skew 500 ppm
Sep 29 18:32:34 dist-test-slave-qcpv krb5kdc[3106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706754, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.207.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:34.732165 3718 init.cc:377] Logged in from keytab as oryx/127.28.207.124@KRBTEST.COM (short username oryx)
I20260929 18:32:34.733271 3718 webserver.cc:460] Webserver started at http://127.28.207.124:45147/ using document root <none> and password file <none>
I20260929 18:32:34.733867 3718 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:34.733961 3718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:34.734210 3718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:34.736603 3718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-2/data/instance:
uuid: "e65dbad386c04dd381b04723f4a864e3"
format_stamp: "Formatted at 2026-09-29 18:32:34 on dist-test-slave-qcpv"
server_key: "c4ffdcbe686b19d7503d5eee086be72a"
server_key_iv: "0273bd5212ac9acaaf4849eb10f9f4f6"
server_key_version: "encryptionkey@0"
I20260929 18:32:34.737269 3718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-2/wal/instance:
uuid: "e65dbad386c04dd381b04723f4a864e3"
format_stamp: "Formatted at 2026-09-29 18:32:34 on dist-test-slave-qcpv"
server_key: "c4ffdcbe686b19d7503d5eee086be72a"
server_key_iv: "0273bd5212ac9acaaf4849eb10f9f4f6"
server_key_version: "encryptionkey@0"
I20260929 18:32:34.741896 3718 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:32:34.745535 3819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:34.747394 3718 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 18:32:34.747545 3718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-2/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-2/wal
uuid: "e65dbad386c04dd381b04723f4a864e3"
format_stamp: "Formatted at 2026-09-29 18:32:34 on dist-test-slave-qcpv"
server_key: "c4ffdcbe686b19d7503d5eee086be72a"
server_key_iv: "0273bd5212ac9acaaf4849eb10f9f4f6"
server_key_version: "encryptionkey@0"
I20260929 18:32:34.747704 3718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:34.761029 3718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:34.764454 3718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:34.764634 3718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:34.772941 3718 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.124:45571
I20260929 18:32:34.773232 3871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.124:45571 every 8 connection(s)
I20260929 18:32:34.774080 3718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-2/data/info.pb
I20260929 18:32:34.778052 3872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:34.780344 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 3718
I20260929 18:32:34.784113 3872 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:32:34 dist-test-slave-qcpv krb5kdc[3106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.207.124@KRBTEST.COM
Sep 29 18:32:34 dist-test-slave-qcpv krb5kdc[3106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706754, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.207.124@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM
I20260929 18:32:34.798547 3190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:34.786788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33640 (local address 127.28.207.126:33133)
0929 18:32:34.786983 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:34.786988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:34.787217 (+ 229us) server_negotiation.cc:407] Connection header received
0929 18:32:34.787376 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:34.787379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:34.787440 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:34.787538 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:34.788496 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.789320 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.790963 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.791334 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.791439 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:34.794792 (+ 3353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:34.794796 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:34.794799 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:34.794830 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:34.796434 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:34.796987 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:34.796991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:34.796993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:34.797079 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:34.797435 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:34.797441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:34.797444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:34.797666 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:34.797784 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:34.798218 (+ 434us) server_negotiation.cc:299] Negotiation successful
0929 18:32:34.798360 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260929 18:32:34.799579 3875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:34.786625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:33133 (local address 127.0.0.1:33640)
0929 18:32:34.787063 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:34.787122 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:34.787264 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:34.787619 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:34.787625 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:34.788111 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:34.788332 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:34.788339 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.789506 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.789509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:34.790687 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:34.790702 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.791533 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.791538 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:34.791692 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:34.792398 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:34.792416 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:34.794612 (+ 2196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:34.796563 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:34.796574 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:34.796585 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:34.796862 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:34.797180 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:34.797185 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:34.797187 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:34.797324 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:34.797780 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:34.797789 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:34.797961 (+ 172us) client_negotiation.cc:769] Sending connection context
0929 18:32:34.798177 (+ 216us) client_negotiation.cc:240] Negotiation successful
0929 18:32:34.798554 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":121,"threads_started":1}
I20260929 18:32:34.799579 3874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:34.782261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52110 (local address 127.28.207.124:45571)
0929 18:32:34.782887 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:34.782912 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:34.782970 (+ 58us) server_negotiation.cc:407] Connection header received
0929 18:32:34.784158 (+ 1188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:34.784185 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:34.784707 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:34.785255 (+ 548us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:34.785923 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.787383 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.788653 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.789490 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.789631 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:34.793493 (+ 3862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:34.793501 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:34.793523 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:34.793546 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:34.796196 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:34.796857 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:34.796863 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:34.796865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:34.797021 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:34.797358 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:34.797367 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:34.797368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:34.797896 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:34.798147 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:34.798967 (+ 820us) server_negotiation.cc:299] Negotiation successful
0929 18:32:34.799139 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":158,"threads_started":1}
I20260929 18:32:34.801409 3872 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.207.125" port: 32877 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:32:34 dist-test-slave-qcpv krb5kdc[3106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706754, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.207.124@KRBTEST.COM for oryx/127.28.207.125@KRBTEST.COM
I20260929 18:32:34.811107 3875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:34.802107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.125:32877 (local address 127.0.0.1:50680)
0929 18:32:34.802264 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:34.802281 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:34.802371 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:34.802834 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:34.802837 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:34.803129 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:34.803253 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:34.803259 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.804477 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.804481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:34.805087 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:34.805094 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.805726 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.805729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:34.805809 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:34.806469 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:34.806480 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:34.807818 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:34.809581 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:34.809588 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:34.809590 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:34.809778 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:34.810051 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:34.810055 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:34.810057 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:34.810138 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:34.810537 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:34.810540 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:34.810635 (+ 95us) client_negotiation.cc:769] Sending connection context
0929 18:32:34.810757 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 18:32:34.810936 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 18:32:34.811129 3876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:34.802258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50680 (local address 127.28.207.125:32877)
0929 18:32:34.802571 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:34.802584 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:34.802599 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:32:34.802645 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:34.802647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:34.802698 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:34.802799 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:34.803495 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.804347 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.805259 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.805520 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.805753 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:34.807990 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:34.807993 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:34.807995 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:34.808040 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:34.809469 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:34.809875 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:34.809880 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:34.809882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:34.809961 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:34.810224 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:34.810230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:34.810231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:34.810434 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:34.810537 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:34.810806 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 18:32:34.810968 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":132,"threads_started":1}
I20260929 18:32:34.812636 3872 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.207.124" port: 45571 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:32:34 dist-test-slave-qcpv krb5kdc[3106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706754, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.207.124@KRBTEST.COM for oryx/127.28.207.124@KRBTEST.COM
I20260929 18:32:34.822801 3874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:34.813322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52118 (local address 127.28.207.124:45571)
0929 18:32:34.813466 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:34.813469 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:34.813512 (+ 43us) server_negotiation.cc:407] Connection header received
0929 18:32:34.813679 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:34.813682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:34.813721 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:34.813807 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:34.814268 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.815217 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.816135 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.816393 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.816516 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:34.818823 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:34.818826 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:34.818828 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:34.818847 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:34.820335 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:34.821017 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:34.821020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:34.821021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:34.821091 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:34.821584 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:34.821607 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:34.821612 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:34.821813 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:34.822084 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:34.822413 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 18:32:34.822633 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":105,"server-negotiator.queue_time_us":49}
I20260929 18:32:34.822801 3875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:34.813245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:45571 (local address 127.0.0.1:52118)
0929 18:32:34.813439 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:34.813465 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:34.813560 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:34.813911 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:34.813917 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:34.814070 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:34.814145 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:34.814151 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.815372 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.815375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:34.815963 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:34.815971 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.816558 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.816561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:34.816639 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:34.817321 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:34.817333 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:34.818694 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:34.820499 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:34.820505 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:34.820509 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:34.820788 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:34.821216 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:34.821219 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:34.821220 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:34.821406 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:34.822081 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:34.822085 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:34.822221 (+ 136us) client_negotiation.cc:769] Sending connection context
0929 18:32:34.822352 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 18:32:34.822564 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 18:32:34.826917 3872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3: Bootstrap starting.
I20260929 18:32:34.829569 3872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:34.830323 3872 log.cc:824] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:34.832700 3872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3: No bootstrap required, opened a new log
I20260929 18:32:34.835973 3872 raft_consensus.cc:348] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 32877 } } peers { permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 45571 } }
I20260929 18:32:34.836231 3872 raft_consensus.cc:374] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:34.836297 3872 raft_consensus.cc:729] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e65dbad386c04dd381b04723f4a864e3, State: Initialized, Role: FOLLOWER
I20260929 18:32:34.836781 3872 consensus_queue.cc:260] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 32877 } } peers { permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 45571 } }
I20260929 18:32:34.837292 3878 sys_catalog.cc:455] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 32877 } } peers { permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 45571 } } }
I20260929 18:32:34.837478 3878 sys_catalog.cc:458] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:32:34.838176 3872 sys_catalog.cc:564] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 18:32:34.844219 3889 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:32:34.844370 3889 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:32:34.848932 3718 master_runner.cc:428] Detected that this master e65dbad386c04dd381b04723f4a864e3 is joining an existing cluster
I20260929 18:32:34.849051 3718 master_runner.cc:432] Initiating AddMaster RPC to add 127.28.207.124:45571
I20260929 18:32:34.853799 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 3213
I20260929 18:32:34.863471 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.65:37323
--local_ip_for_outbound_sockets=127.28.207.65
--tserver_master_addrs=127.28.207.126:33133
--webserver_port=34767
--webserver_interface=127.28.207.65
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.65.keytab
--principal=oryx/127.28.207.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:42331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.207.126:33133,127.28.207.125:32877
--tserver_master_addrs=127.28.207.126:33133,127.28.207.125:32877,127.28.207.124:45571 with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:32:34.868791 3876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:34.852362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50692 (local address 127.28.207.125:32877)
0929 18:32:34.852554 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:34.852561 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:34.853426 (+ 865us) server_negotiation.cc:407] Connection header received
0929 18:32:34.853678 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:34.853683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:34.853759 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:34.853906 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:34.856383 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.857491 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.859788 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.860364 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.860539 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:34.863930 (+ 3391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:34.863935 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:34.863939 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:34.863980 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:34.866101 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:34.866741 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:34.866745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:34.866748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:34.866843 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:34.867445 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:34.867449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:34.867451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:34.867753 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:34.867853 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:34.868370 (+ 517us) server_negotiation.cc:299] Negotiation successful
0929 18:32:34.868565 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 18:32:34.869544 3895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:34.852730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.125:32877 (local address 127.0.0.1:50692)
0929 18:32:34.853321 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:34.853340 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:34.853516 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:34.855573 (+ 2057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:34.855581 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:34.855949 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:34.856168 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:34.856177 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.857658 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.857663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:34.859397 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:34.859414 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.860574 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.860578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:34.860702 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:34.862744 (+ 2042us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:34.862763 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:34.863665 (+ 902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:34.866275 (+ 2610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:34.866278 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:34.866281 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:34.866605 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:34.867162 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:34.867166 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:34.867169 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:34.867302 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:34.867917 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:34.867921 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:34.868053 (+ 132us) client_negotiation.cc:769] Sending connection context
0929 18:32:34.869104 (+ 1051us) client_negotiation.cc:240] Negotiation successful
0929 18:32:34.869319 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":160,"threads_started":1}
I20260929 18:32:34.893636 3874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:34.852497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52134 (local address 127.28.207.124:45571)
0929 18:32:34.852620 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:34.852625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:34.861269 (+ 8644us) server_negotiation.cc:407] Connection header received
0929 18:32:34.861323 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:34.861326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:34.861386 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:34.861499 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:34.883957 (+ 22458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.884807 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.886352 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.886758 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.886898 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:34.888760 (+ 1862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:34.888765 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:34.888768 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:34.888793 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:34.891700 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:34.892274 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:34.892278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:34.892280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:34.892359 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:34.892627 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:34.892630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:34.892632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:34.892877 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:34.892960 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:34.893250 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 18:32:34.893422 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:32:34.893767 3190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:34.852247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33654 (local address 127.28.207.126:33133)
0929 18:32:34.852623 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:34.852629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:34.861518 (+ 8889us) server_negotiation.cc:407] Connection header received
0929 18:32:34.861722 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:34.861728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:34.861799 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:34.861865 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:34.883999 (+ 22134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.884971 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.886422 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.886781 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.886920 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:34.888967 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:34.888972 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:34.888974 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:34.889004 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:34.891428 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:34.892250 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:34.892255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:34.892256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:34.892365 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:34.892682 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:34.892686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:34.892687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:34.892932 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:34.893014 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:34.893366 (+ 352us) server_negotiation.cc:299] Negotiation successful
0929 18:32:34.893500 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250}
I20260929 18:32:34.894232 3896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:34.852790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:45571 (local address 127.0.0.1:52134)
0929 18:32:34.860938 (+ 8148us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:34.860960 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:34.861114 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:34.861639 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:34.861643 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:34.883418 (+ 21775us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:34.883696 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:34.883716 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.884952 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.884955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:34.885923 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:34.885933 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.887120 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.887124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:34.887230 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:34.887864 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:34.887883 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:34.888568 (+ 685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:34.891846 (+ 3278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:34.891850 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:34.891852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:34.892171 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:34.892453 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:34.892456 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:34.892458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:34.892533 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:34.893003 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:34.893006 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:34.893086 (+ 80us) client_negotiation.cc:769] Sending connection context
0929 18:32:34.893909 (+ 823us) client_negotiation.cc:240] Negotiation successful
0929 18:32:34.894065 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8010,"thread_start_us":160,"threads_started":1}
I20260929 18:32:34.895344 3897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:34.853195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:33133 (local address 127.0.0.1:33654)
0929 18:32:34.861414 (+ 8219us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:34.861430 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:34.861598 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:34.862074 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:34.862080 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:34.883498 (+ 21418us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:34.883744 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:34.883760 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.885190 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.885194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:34.886261 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:34.886272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.887253 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.887257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:34.887361 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:34.888112 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:34.888130 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:34.888763 (+ 633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:34.891680 (+ 2917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:34.891685 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:34.891687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:34.892111 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:34.892483 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:34.892486 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:34.892488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:34.892581 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:34.893071 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:34.893074 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:34.893178 (+ 104us) client_negotiation.cc:769] Sending connection context
0929 18:32:34.894596 (+ 1418us) client_negotiation.cc:240] Negotiation successful
0929 18:32:34.894792 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8120,"thread_start_us":99,"threads_started":1}
I20260929 18:32:34.907518 3878 raft_consensus.cc:482] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:32:34.907734 3878 raft_consensus.cc:504] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 32877 } } peers { permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 45571 } }
Sep 29 18:32:34 dist-test-slave-qcpv krb5kdc[3106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706753, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.207.126@KRBTEST.COM for oryx/127.28.207.124@KRBTEST.COM
I20260929 18:32:34.909188 3878 leader_election.cc:288] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1250cda9f69445ecadffa350f7d1790c (127.28.207.126:33133), 9f296fd5b96e4975b59022170377b843 (127.28.207.125:32877)
I20260929 18:32:34.910420 3161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e65dbad386c04dd381b04723f4a864e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1250cda9f69445ecadffa350f7d1790c" is_pre_election: true
I20260929 18:32:34.912063 3764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e65dbad386c04dd381b04723f4a864e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f296fd5b96e4975b59022170377b843" is_pre_election: true
I20260929 18:32:34.912276 3764 raft_consensus.cc:1799] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [term 1 LEARNER]: Handling vote request from an unknown peer e65dbad386c04dd381b04723f4a864e3
I20260929 18:32:34.912402 3764 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate e65dbad386c04dd381b04723f4a864e3 for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260929 18:32:34.912890 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:34.900863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:45571 (local address 127.0.0.1:52136)
0929 18:32:34.901113 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:34.901130 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:34.901294 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:34.901739 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:34.901744 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:34.901986 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:34.902138 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:34.902146 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.903330 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.903335 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:34.904353 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:34.904367 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.905171 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.905178 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:34.905305 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:34.906190 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:34.906393 (+ 203us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:34.908497 (+ 2104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:34.910908 (+ 2411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:34.910912 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:34.910915 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:34.911195 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:34.911533 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:34.911537 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:34.911539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:34.911642 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:34.912136 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:34.912142 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:34.912262 (+ 120us) client_negotiation.cc:769] Sending connection context
0929 18:32:34.912366 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 18:32:34.912561 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140}
I20260929 18:32:34.913071 3821 leader_election.cc:302] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e65dbad386c04dd381b04723f4a864e3; no voters: 1250cda9f69445ecadffa350f7d1790c, 9f296fd5b96e4975b59022170377b843
I20260929 18:32:34.913878 3874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:34.900988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52136 (local address 127.28.207.124:45571)
0929 18:32:34.901120 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:34.901125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:34.901213 (+ 88us) server_negotiation.cc:407] Connection header received
0929 18:32:34.901461 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:34.901464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:34.901527 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:34.901640 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:34.902281 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.903117 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.904629 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.905003 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.905128 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:34.908783 (+ 3655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:34.908789 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:34.908792 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:34.908830 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:34.910716 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:34.911331 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:34.911335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:34.911337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:34.911426 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:34.911752 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:34.911758 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:34.911761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:34.912024 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:34.913434 (+ 1410us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:34.913527 (+ 93us) server_negotiation.cc:299] Negotiation successful
0929 18:32:34.913696 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:32:34.914006 3878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:34.915654 3878 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 18:32:34.917623 3151 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.28.207.124:45571: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e65dbad386c04dd381b04723f4a864e3"
member_type: NON_VOTER
last_known_addr {
host: "127.28.207.124"
port: 45571
}
attrs {
promote: true
}
}
dest_uuid: "1250cda9f69445ecadffa350f7d1790c"
cas_config_opid_index: 5
I20260929 18:32:34.918290 3151 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: NON_VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: true } } peers { permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: NON_VOTER last_known_addr { host: "127.28.207.124" port: 45571 } attrs { promote: true } }
I20260929 18:32:34.919934 3712 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e65dbad386c04dd381b04723f4a864e3 (127.28.207.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: NON_VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: true } } peers { permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: NON_VOTER last_known_addr { host: "127.28.207.124" port: 45571 } attrs { promote: true } } }
I20260929 18:32:34.920997 3846 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [term 1 FOLLOWER]: Refusing update from remote peer 1250cda9f69445ecadffa350f7d1790c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 18:32:34.921561 3715 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: NON_VOTER last_known_addr { host: "127.28.207.124" port: 45571 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:32:34.922971 3712 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.28.207.124:45571
I20260929 18:32:34.923254 3712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1250cda9f69445ecadffa350f7d1790c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: NON_VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: true } } peers { permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: NON_VOTER last_known_addr { host: "127.28.207.124" port: 45571 } attrs { promote: true } } }
I20260929 18:32:34.923375 3712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:34.924527 3718 master.cc:537] Master@127.28.207.124:45571 shutting down...
I20260929 18:32:34.926646 3846 raft_consensus.cc:858] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 32877 } } peers { permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 45571 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: NON_VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: NON_VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: true } } peers { permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: NON_VOTER last_known_addr { host: "127.28.207.124" port: 45571 } attrs { promote: true } }
W20260929 18:32:34.926873 3846 raft_consensus.cc:1565] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 32877 } } peers { permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 45571 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: NON_VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: NON_VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: true } } peers { permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: NON_VOTER last_known_addr { host: "127.28.207.124" port: 45571 } attrs { promote: true } }
I20260929 18:32:34.928705 3846 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 9f296fd5b96e4975b59022170377b843 (127.28.207.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: NON_VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: true } } }
I20260929 18:32:34.931058 3878 sys_catalog.cc:455] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1250cda9f69445ecadffa350f7d1790c. Latest consensus state: current_term: 1 leader_uuid: "1250cda9f69445ecadffa350f7d1790c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: NON_VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: true } } }
I20260929 18:32:34.931170 3878 sys_catalog.cc:458] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260929 18:32:34.931252 3878 sys_catalog.cc:455] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1250cda9f69445ecadffa350f7d1790c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: NON_VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: true } } }
I20260929 18:32:34.931301 3878 sys_catalog.cc:458] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260929 18:32:34.931352 3878 sys_catalog.cc:455] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1250cda9f69445ecadffa350f7d1790c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: NON_VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: true } } }
I20260929 18:32:34.931402 3878 sys_catalog.cc:458] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260929 18:32:34.933854 3899 mvcc.cc:205] Tried to move back new op lower bound from 7334734861907607552 to 7334734861657800704. Current Snapshot: MvccSnapshot[applied={T|T < 7334734861702610944}]
I20260929 18:32:34.935899 3876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:34.920263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50708 (local address 127.28.207.125:32877)
0929 18:32:34.921940 (+ 1677us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:34.921946 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:34.929977 (+ 8031us) server_negotiation.cc:407] Connection header received
0929 18:32:34.930257 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:34.930261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:34.930343 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:34.930422 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:34.930930 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.931815 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.933679 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.934403 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.934516 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:34.934695 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:34.935614 (+ 919us) server_negotiation.cc:299] Negotiation successful
0929 18:32:34.935730 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1553}
I20260929 18:32:34.936787 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:34.919792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.125:32877 (local address 127.0.0.1:50708)
0929 18:32:34.929826 (+ 10034us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:34.929850 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:34.930106 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:34.930620 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:34.930623 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:34.930640 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:34.930762 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:34.930769 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.932126 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.932130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:34.933522 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:34.933536 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.934949 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.934954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:34.935063 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:34.935304 (+ 241us) client_negotiation.cc:769] Sending connection context
0929 18:32:34.936486 (+ 1182us) client_negotiation.cc:240] Negotiation successful
0929 18:32:34.936640 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9915}
I20260929 18:32:34.937664 3764 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [term 1 LEARNER]: Refusing update from remote peer 1250cda9f69445ecadffa350f7d1790c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 18:32:34.938301 3712 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: NON_VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 18:32:34.941644 3764 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e65dbad386c04dd381b04723f4a864e3 (127.28.207.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: NON_VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: true } } peers { permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: NON_VOTER last_known_addr { host: "127.28.207.124" port: 45571 } attrs { promote: true } } }
W20260929 18:32:34.942509 3135 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c -> Peer e65dbad386c04dd381b04723f4a864e3 (127.28.207.124:45571): Couldn't send request to peer e65dbad386c04dd381b04723f4a864e3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 18:32:34.942734 3715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1250cda9f69445ecadffa350f7d1790c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: NON_VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: true } } peers { permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: NON_VOTER last_known_addr { host: "127.28.207.124" port: 45571 } attrs { promote: true } } }
I20260929 18:32:34.942893 3715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:34.944063 3718 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260929 18:32:34.944495 3718 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260929 18:32:34.944607 3718 tablet_replica.cc:330] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3: stopping tablet replica
I20260929 18:32:34.945277 3793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1250cda9f69445ecadffa350f7d1790c. Latest consensus state: current_term: 1 leader_uuid: "1250cda9f69445ecadffa350f7d1790c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: NON_VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: true } } peers { permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: NON_VOTER last_known_addr { host: "127.28.207.124" port: 45571 } attrs { promote: true } } }
I20260929 18:32:34.945451 3793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [sys.catalog]: This master's current role is: LEARNER
I20260929 18:32:34.945592 3793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1250cda9f69445ecadffa350f7d1790c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: NON_VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: true } } peers { permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: NON_VOTER last_known_addr { host: "127.28.207.124" port: 45571 } attrs { promote: true } } }
I20260929 18:32:34.945681 3793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [sys.catalog]: This master's current role is: LEARNER
I20260929 18:32:34.947875 3712 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c: attempting to promote NON_VOTER 9f296fd5b96e4975b59022170377b843 to VOTER
I20260929 18:32:34.948372 3712 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: false } } peers { permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: NON_VOTER last_known_addr { host: "127.28.207.124" port: 45571 } attrs { promote: true } }
I20260929 18:32:34.950104 3764 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [term 1 LEARNER]: Refusing update from remote peer 1250cda9f69445ecadffa350f7d1790c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 18:32:34.950778 3714 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260929 18:32:34.951651 3135 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c -> Peer e65dbad386c04dd381b04723f4a864e3 (127.28.207.124:45571): Couldn't send request to peer e65dbad386c04dd381b04723f4a864e3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 18:32:34.953649 3715 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9f296fd5b96e4975b59022170377b843 (127.28.207.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: false } } peers { permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: NON_VOTER last_known_addr { host: "127.28.207.124" port: 45571 } attrs { promote: true } } }
I20260929 18:32:34.954358 3764 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9f296fd5b96e4975b59022170377b843 (127.28.207.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: false } } peers { permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: NON_VOTER last_known_addr { host: "127.28.207.124" port: 45571 } attrs { promote: true } } }
I20260929 18:32:34.955394 3715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1250cda9f69445ecadffa350f7d1790c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: false } } peers { permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: NON_VOTER last_known_addr { host: "127.28.207.124" port: 45571 } attrs { promote: true } } }
I20260929 18:32:34.955495 3715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:34.955611 3715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1250cda9f69445ecadffa350f7d1790c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: false } } peers { permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: NON_VOTER last_known_addr { host: "127.28.207.124" port: 45571 } attrs { promote: true } } }
I20260929 18:32:34.955673 3715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:34.957357 3907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1250cda9f69445ecadffa350f7d1790c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: false } } peers { permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: NON_VOTER last_known_addr { host: "127.28.207.124" port: 45571 } attrs { promote: true } } }
I20260929 18:32:34.957518 3907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:32:34.968719 3718 master.cc:552] Master@127.28.207.124:45571 shutdown complete.
I20260929 18:32:34.968813 3718 master_runner.cc:305] Clearing existing system tablet
I20260929 18:32:34.969180 3476 heartbeater.cc:503] Master 127.28.207.126:33133 was elected leader, sending a full tablet report...
I20260929 18:32:34.969631 3718 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:34.969707 3718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:34.971531 3718 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 18:32:34.972651 3909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:34.972957 3718 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:34.973096 3718 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:32:34.973142 3718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-2/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-2/wal
uuid: "e65dbad386c04dd381b04723f4a864e3"
format_stamp: "Formatted at 2026-09-29 18:32:34 on dist-test-slave-qcpv"
server_key: "c4ffdcbe686b19d7503d5eee086be72a"
server_key_iv: "0273bd5212ac9acaaf4849eb10f9f4f6"
server_key_version: "encryptionkey@0"
I20260929 18:32:34.976191 3718 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:32:34.979597 3718 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 18:32:34.979763 3718 log.cc:1196] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 18:32:34.980131 3718 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3: Deleting consensus metadata
I20260929 18:32:34.980365 3718 master_runner.cc:315] Copying system tablet from 127.28.207.126:33133
I20260929 18:32:34.982488 3718 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3: tablet copy: Beginning tablet copy session from remote peer at address 127.28.207.126:33133
I20260929 18:32:34.994704 3915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:34.983327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:33133 (local address 127.0.0.1:33664)
0929 18:32:34.983769 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:34.983787 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:34.983973 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:34.984449 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:34.984452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:34.984744 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:34.984886 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:34.984893 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.986090 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.986180 (+ 90us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:34.987267 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:34.987284 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.988057 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.988062 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:34.988214 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:34.989003 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:34.989021 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:34.989580 (+ 559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:34.992523 (+ 2943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:34.992527 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:34.992530 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:34.992876 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:34.993270 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:34.993273 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:34.993278 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:34.993380 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:34.993979 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:34.993983 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:34.994123 (+ 140us) client_negotiation.cc:769] Sending connection context
0929 18:32:34.994248 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 18:32:34.994425 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":140,"threads_started":1}
I20260929 18:32:34.994760 3190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:34.983415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33664 (local address 127.28.207.126:33133)
0929 18:32:34.983585 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:34.983590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:34.983950 (+ 360us) server_negotiation.cc:407] Connection header received
0929 18:32:34.984108 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:34.984122 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:34.984209 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:34.984343 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:34.985042 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.985905 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.987527 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:34.987882 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:34.988011 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:34.989789 (+ 1778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:34.989804 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:34.989807 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:34.989843 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:34.992343 (+ 2500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:34.993027 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:34.993032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:34.993034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:34.993153 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:34.993567 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:34.993571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:34.993572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:34.993860 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:34.993976 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:34.994338 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 18:32:34.994497 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 18:32:34.996003 3171 tablet_copy_service.cc:140] P 1250cda9f69445ecadffa350f7d1790c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e65dbad386c04dd381b04723f4a864e3 ({username='oryx', principal='oryx/127.28.207.124@KRBTEST.COM'} at 127.0.0.1:33664)
I20260929 18:32:34.996153 3171 tablet_copy_service.cc:161] P 1250cda9f69445ecadffa350f7d1790c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e65dbad386c04dd381b04723f4a864e3 at {username='oryx', principal='oryx/127.28.207.124@KRBTEST.COM'} at 127.0.0.1:33664: session id = e65dbad386c04dd381b04723f4a864e3-00000000000000000000000000000000
I20260929 18:32:34.997330 3171 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 18:32:34.999225 3718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 18:32:35.002030 3718 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3: tablet copy: Starting download of 0 data blocks...
I20260929 18:32:35.002249 3718 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3: tablet copy: Starting download of 1 WAL segments...
I20260929 18:32:35.004115 3718 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 18:32:35.006196 3171 tablet_copy_service.cc:342] P 1250cda9f69445ecadffa350f7d1790c: Request end of tablet copy session e65dbad386c04dd381b04723f4a864e3-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.28.207.124@KRBTEST.COM'} at 127.0.0.1:33664
I20260929 18:32:35.006292 3171 tablet_copy_service.cc:433] P 1250cda9f69445ecadffa350f7d1790c: ending tablet copy session e65dbad386c04dd381b04723f4a864e3-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e65dbad386c04dd381b04723f4a864e3
I20260929 18:32:35.015405 3718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:35.017463 3916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:35.017517 3917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:35.017771 3919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:35.018399 3718 server_base.cc:1034] running on GCE node
I20260929 18:32:35.018620 3718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:35.019035 3718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:35.020733 3718 hybrid_clock.cc:648] HybridClock initialized: now 1790706755020723 us; error 34 us; skew 500 ppm
Sep 29 18:32:35 dist-test-slave-qcpv krb5kdc[3106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706755, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.207.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:35.022893 3718 init.cc:377] Logged in from keytab as oryx/127.28.207.124@KRBTEST.COM (short username oryx)
I20260929 18:32:35.023523 3718 webserver.cc:460] Webserver started at http://127.28.207.124:37127/ using document root <none> and password file <none>
I20260929 18:32:35.023805 3718 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:35.023908 3718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:35.025380 3718 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 18:32:35.027055 3929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:35.027787 3718 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 18:32:35.027872 3718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-2/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-2/wal
uuid: "e65dbad386c04dd381b04723f4a864e3"
format_stamp: "Formatted at 2026-09-29 18:32:34 on dist-test-slave-qcpv"
server_key: "c4ffdcbe686b19d7503d5eee086be72a"
server_key_iv: "0273bd5212ac9acaaf4849eb10f9f4f6"
server_key_version: "encryptionkey@0"
I20260929 18:32:35.027949 3718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260929 18:32:35.032894 3898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:35.033281 3898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:35.033342 3898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:35.039597 3898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:35.039697 3898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:35.039786 3898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.65
I20260929 18:32:35.049093 3898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:42331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.65.keytab
--principal=oryx/127.28.207.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.65:37323
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.207.65
--webserver_port=34767
--tserver_master_addrs=127.28.207.126:33133,127.28.207.125:32877,127.28.207.124:45571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.65
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:35.050886 3898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:35.052233 3898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:32:35.057232 3718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:35.060220 3718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:35.065752 3718 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.124:45571
I20260929 18:32:35.066431 3718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/master-2/data/info.pb
I20260929 18:32:35.066995 3986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.124:45571 every 8 connection(s)
W20260929 18:32:35.067466 3938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:35.067474 3936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:35.068042 3935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:35.068387 3898 server_base.cc:1034] running on GCE node
I20260929 18:32:35.068990 3898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:35.069638 3988 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:32:35.069797 3898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:35.071310 3988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3: Bootstrap starting.
I20260929 18:32:35.073132 3898 hybrid_clock.cc:648] HybridClock initialized: now 1790706755073084 us; error 61 us; skew 500 ppm
Sep 29 18:32:35 dist-test-slave-qcpv krb5kdc[3106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706755, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.207.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:35.076846 3898 init.cc:377] Logged in from keytab as oryx/127.28.207.65@KRBTEST.COM (short username oryx)
I20260929 18:32:35.078132 3898 webserver.cc:460] Webserver started at http://127.28.207.65:34767/ using document root <none> and password file <none>
I20260929 18:32:35.078743 3898 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:35.078905 3898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:35.083335 3898 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:32:35.085429 3988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:32:35.085847 3988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3: Bootstrap complete.
I20260929 18:32:35.086056 3996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:35.086562 3988 raft_consensus.cc:348] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: false } } peers { permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: NON_VOTER last_known_addr { host: "127.28.207.124" port: 45571 } attrs { promote: true } }
I20260929 18:32:35.086733 3988 raft_consensus.cc:729] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e65dbad386c04dd381b04723f4a864e3, State: Initialized, Role: LEARNER
I20260929 18:32:35.086879 3988 consensus_queue.cc:260] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: false } } peers { permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: NON_VOTER last_known_addr { host: "127.28.207.124" port: 45571 } attrs { promote: true } }
I20260929 18:32:35.087448 3988 sys_catalog.cc:564] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:32:35.087865 3898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:32:35.088013 3898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-0/wal
uuid: "52f601bc57eb4420854cb9073fdd9caa"
format_stamp: "Formatted at 2026-09-29 18:32:33 on dist-test-slave-qcpv"
server_key: "33cc506226a82bc52db3fcd9ee5ddcf9"
server_key_iv: "dd84b2fd57b1da396f612f4d49173831"
server_key_version: "encryptionkey@0"
I20260929 18:32:35.088613 3898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:35.091696 3997 sys_catalog.cc:455] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: false } } peers { permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: NON_VOTER last_known_addr { host: "127.28.207.124" port: 45571 } attrs { promote: true } } }
I20260929 18:32:35.091845 3997 sys_catalog.cc:458] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [sys.catalog]: This master's current role is: LEARNER
I20260929 18:32:35.093458 4008 catalog_manager.cc:1259] Loaded cluster ID: 41ad7488d0df4de6922e7f2e0903f63a
I20260929 18:32:35.093513 4008 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3: loading cluster ID for follower catalog manager: success
I20260929 18:32:35.095525 4008 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3: acquiring CA information for follower catalog manager: success
I20260929 18:32:35.096808 4008 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:32:35.111960 3898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:35.115980 3898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:35.116204 3898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:35.116885 3898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:32:35.117791 3898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:35.117870 3898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:35.117937 3898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:35.117990 3898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:35.130519 3898 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.65:37323
I20260929 18:32:35.130540 4121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.65:37323 every 8 connection(s)
I20260929 18:32:35.131811 3898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
I20260929 18:32:35.134593 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 3898
I20260929 18:32:35.134812 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 3347
I20260929 18:32:35.143515 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.66:44677
--local_ip_for_outbound_sockets=127.28.207.66
--tserver_master_addrs=127.28.207.126:33133
--webserver_port=39381
--webserver_interface=127.28.207.66
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.66.keytab
--principal=oryx/127.28.207.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:42331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.207.126:33133,127.28.207.125:32877
--tserver_master_addrs=127.28.207.126:33133,127.28.207.125:32877,127.28.207.124:45571 with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:32:35 dist-test-slave-qcpv krb5kdc[3106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706755, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.207.65@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM
Sep 29 18:32:35 dist-test-slave-qcpv krb5kdc[3106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706755, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.207.65@KRBTEST.COM for oryx/127.28.207.125@KRBTEST.COM
Sep 29 18:32:35 dist-test-slave-qcpv krb5kdc[3106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706755, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.207.65@KRBTEST.COM for oryx/127.28.207.124@KRBTEST.COM
I20260929 18:32:35.158574 3190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:35.136093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:41291 (local address 127.28.207.126:33133)
0929 18:32:35.136216 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:35.136220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:35.139235 (+ 3015us) server_negotiation.cc:407] Connection header received
0929 18:32:35.139444 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:35.139448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:35.139518 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:35.139582 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:35.141882 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.142986 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.145922 (+ 2936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.146249 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.146527 (+ 278us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:35.152336 (+ 5809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:35.152341 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:35.152345 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:35.152390 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:35.155139 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:35.156021 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:35.156028 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:35.156032 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:35.156153 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:35.156735 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:35.156738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:35.156741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:35.157030 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:35.157122 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:35.158193 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0929 18:32:35.158358 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:32:35.158591 4129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:35.137750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:53179 (local address 127.28.207.124:45571)
0929 18:32:35.138069 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:35.138074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:35.138261 (+ 187us) server_negotiation.cc:407] Connection header received
0929 18:32:35.139200 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:35.139207 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:35.139282 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:35.139386 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:35.141876 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.143491 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.145810 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.146217 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.146508 (+ 291us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:35.152239 (+ 5731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:35.152248 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:35.152256 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:35.152304 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:35.155402 (+ 3098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:35.156078 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:35.156084 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:35.156089 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:35.156186 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:35.156735 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:35.156738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:35.156741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:35.157030 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:35.157114 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:35.158177 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0929 18:32:35.158363 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":123,"threads_started":1}
I20260929 18:32:35.158613 3876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:35.135071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:49875 (local address 127.28.207.125:32877)
0929 18:32:35.135263 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:35.135268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:35.137407 (+ 2139us) server_negotiation.cc:407] Connection header received
0929 18:32:35.139044 (+ 1637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:35.139051 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:35.139131 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:35.139196 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:35.141883 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.142911 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.145538 (+ 2627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.145807 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.145948 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:35.152441 (+ 6493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:35.152448 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:35.152454 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:35.152495 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:35.154898 (+ 2403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:35.156033 (+ 1135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:35.156039 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:35.156043 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:35.156153 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:35.156766 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:35.156773 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:35.156777 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:35.157072 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:35.157175 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:35.158185 (+ 1010us) server_negotiation.cc:299] Negotiation successful
0929 18:32:35.158360 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 18:32:35.160939 4126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:35.135264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:45571 (local address 127.28.207.65:53179)
0929 18:32:35.138119 (+ 2855us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:35.138137 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:35.138683 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:35.139602 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:35.139616 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:35.140561 (+ 945us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:35.141658 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:35.141686 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.143801 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.143809 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:35.145689 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:35.145699 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.146686 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.146689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:35.146880 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:35.147745 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:35.147765 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:35.151793 (+ 4028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:35.155591 (+ 3798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:35.155605 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:35.155611 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:35.155958 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:35.156307 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:35.156311 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:35.156312 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:35.156620 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:35.157216 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:35.157226 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:35.157794 (+ 568us) client_negotiation.cc:769] Sending connection context
0929 18:32:35.159669 (+ 1875us) client_negotiation.cc:240] Negotiation successful
0929 18:32:35.159817 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2723,"mutex_wait_us":854,"spinlock_wait_cycles":52608,"thread_start_us":141,"threads_started":1}
I20260929 18:32:35.161245 4128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:35.135264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.125:32877 (local address 127.28.207.65:49875)
0929 18:32:35.137165 (+ 1901us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:35.137231 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:35.138666 (+ 1435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:35.139603 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:35.139618 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:35.140579 (+ 961us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:35.141676 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:35.141694 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.143230 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.143236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:35.145366 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:35.145381 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.146026 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.146031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:35.146483 (+ 452us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:35.148216 (+ 1733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:35.148239 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:35.152222 (+ 3983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:35.155116 (+ 2894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:35.155126 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:35.155152 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:35.155877 (+ 725us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:35.156372 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:35.156377 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:35.156382 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:35.156616 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:35.157262 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:35.157268 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:35.157799 (+ 531us) client_negotiation.cc:769] Sending connection context
0929 18:32:35.158888 (+ 1089us) client_negotiation.cc:240] Negotiation successful
0929 18:32:35.159449 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1635,"mutex_wait_us":268,"spinlock_wait_cycles":20608,"thread_start_us":751,"threads_started":1}
I20260929 18:32:35.160939 4127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:35.135352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:33133 (local address 127.28.207.65:41291)
0929 18:32:35.139124 (+ 3772us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:35.139141 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:35.139308 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:35.139762 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:35.139765 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:35.140716 (+ 951us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:35.141658 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:35.141686 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.143190 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.143194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:35.145372 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:35.145391 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.146539 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.146543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:35.146675 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:35.147937 (+ 1262us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:35.147988 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:35.151831 (+ 3843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:35.155333 (+ 3502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:35.155339 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:35.155342 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:35.155874 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:35.156297 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:35.156301 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:35.156303 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:35.156598 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:35.157216 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:35.157226 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:35.157794 (+ 568us) client_negotiation.cc:769] Sending connection context
0929 18:32:35.158813 (+ 1019us) client_negotiation.cc:240] Negotiation successful
0929 18:32:35.159570 (+ 757us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3705,"mutex_wait_us":1838,"spinlock_wait_cycles":49024,"thread_start_us":170,"threads_started":1}
I20260929 18:32:35.162994 4124 heartbeater.cc:347] Connected to a master server at 127.28.207.125:32877
I20260929 18:32:35.163064 4122 heartbeater.cc:347] Connected to a master server at 127.28.207.124:45571
I20260929 18:32:35.163064 4123 heartbeater.cc:347] Connected to a master server at 127.28.207.126:33133
I20260929 18:32:35.163305 4122 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:35.163304 4124 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:35.163305 4123 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:35.164152 4124 heartbeater.cc:511] Master 127.28.207.125:32877 requested a full tablet report, sending...
I20260929 18:32:35.164898 4123 heartbeater.cc:511] Master 127.28.207.126:33133 requested a full tablet report, sending...
I20260929 18:32:35.164152 4122 heartbeater.cc:511] Master 127.28.207.124:45571 requested a full tablet report, sending...
I20260929 18:32:35.166096 3746 ts_manager.cc:194] Registered new tserver with Master: 52f601bc57eb4420854cb9073fdd9caa (127.28.207.65:37323)
I20260929 18:32:35.166134 3151 ts_manager.cc:194] Re-registered known tserver with Master: 52f601bc57eb4420854cb9073fdd9caa (127.28.207.65:37323)
I20260929 18:32:35.166383 3951 ts_manager.cc:194] Registered new tserver with Master: 52f601bc57eb4420854cb9073fdd9caa (127.28.207.65:37323)
I20260929 18:32:35.167140 3151 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.207.65@KRBTEST.COM'} at 127.28.207.65:41291
I20260929 18:32:35.211825 3610 heartbeater.cc:503] Master 127.28.207.126:33133 was elected leader, sending a full tablet report...
W20260929 18:32:35.268689 4130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:35.269023 4130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:35.269109 4130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:35.273474 4130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:35.273572 4130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:35.273663 4130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.66
I20260929 18:32:35.279668 4130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:42331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.66.keytab
--principal=oryx/127.28.207.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.66:44677
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.207.66
--webserver_port=39381
--tserver_master_addrs=127.28.207.126:33133,127.28.207.125:32877,127.28.207.124:45571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.66
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:35.280916 4130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:35.282047 4130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:32:35.293723 4130 server_base.cc:1034] running on GCE node
W20260929 18:32:35.293665 4136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:35.293712 4138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:35.293671 4135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:35.294528 4130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:35.295418 4130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:35.296617 4130 hybrid_clock.cc:648] HybridClock initialized: now 1790706755296573 us; error 63 us; skew 500 ppm
Sep 29 18:32:35 dist-test-slave-qcpv krb5kdc[3106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706755, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.207.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:35.300973 4130 init.cc:377] Logged in from keytab as oryx/127.28.207.66@KRBTEST.COM (short username oryx)
I20260929 18:32:35.302333 4130 webserver.cc:460] Webserver started at http://127.28.207.66:39381/ using document root <none> and password file <none>
I20260929 18:32:35.303172 4130 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:35.303306 4130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:35.308583 4130 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:32:35.312084 4145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:35.314199 4130 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:32:35.314430 4130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-1/wal
uuid: "af2018baa42e42b9b287a44255461376"
format_stamp: "Formatted at 2026-09-29 18:32:33 on dist-test-slave-qcpv"
server_key: "202346058d07fcf6406cb7b3d08a752c"
server_key_iv: "c250200120a9c76f70f01aef047875c1"
server_key_version: "encryptionkey@0"
I20260929 18:32:35.315434 4130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:35.348943 4130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:35.353576 4130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:35.353950 4130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:35.354821 4130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:32:35.356120 4130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:35.356185 4130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:35.356236 4130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:35.356309 4130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:35.369483 4130 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.66:44677
I20260929 18:32:35.369505 4258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.66:44677 every 8 connection(s)
I20260929 18:32:35.370955 4130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
I20260929 18:32:35.375061 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 4130
I20260929 18:32:35.375263 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 3481
I20260929 18:32:35.384011 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.67:36657
--local_ip_for_outbound_sockets=127.28.207.67
--tserver_master_addrs=127.28.207.126:33133
--webserver_port=35729
--webserver_interface=127.28.207.67
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.67.keytab
--principal=oryx/127.28.207.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:42331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.28.207.126:33133,127.28.207.125:32877
--tserver_master_addrs=127.28.207.126:33133,127.28.207.125:32877,127.28.207.124:45571 with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:32:35 dist-test-slave-qcpv krb5kdc[3106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706755, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.207.66@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM
Sep 29 18:32:35 dist-test-slave-qcpv krb5kdc[3106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706755, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.207.66@KRBTEST.COM for oryx/127.28.207.124@KRBTEST.COM
Sep 29 18:32:35 dist-test-slave-qcpv krb5kdc[3106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706755, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.207.66@KRBTEST.COM for oryx/127.28.207.125@KRBTEST.COM
I20260929 18:32:35.394008 3190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:35.373632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:47751 (local address 127.28.207.126:33133)
0929 18:32:35.373752 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:35.373758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:35.379868 (+ 6110us) server_negotiation.cc:407] Connection header received
0929 18:32:35.380012 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:35.380016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:35.380091 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:35.380180 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:35.381031 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.381892 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.383249 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.383598 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.383690 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:35.388159 (+ 4469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:35.388164 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:35.388168 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:35.388210 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:35.391600 (+ 3390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:35.392215 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:35.392219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:35.392221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:35.392320 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:35.392649 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:35.392654 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:35.392659 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:35.392958 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:35.393041 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:35.393629 (+ 588us) server_negotiation.cc:299] Negotiation successful
0929 18:32:35.393801 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:32:35.394102 4129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:35.373528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:59999 (local address 127.28.207.124:45571)
0929 18:32:35.373695 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:35.373702 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:35.376421 (+ 2719us) server_negotiation.cc:407] Connection header received
0929 18:32:35.377482 (+ 1061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:35.377488 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:35.377560 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:35.377667 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:35.379628 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.380532 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.382786 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.383272 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.383357 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:35.388623 (+ 5266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:35.388630 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:35.388637 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:35.388674 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:35.390891 (+ 2217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:35.391759 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:35.391766 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:35.391769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:35.391866 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:35.392351 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:35.392354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:35.392356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:35.392647 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:35.392748 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:35.393667 (+ 919us) server_negotiation.cc:299] Negotiation successful
0929 18:32:35.393848 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 18:32:35.395344 3876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:35.374148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:58881 (local address 127.28.207.125:32877)
0929 18:32:35.374285 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:35.374288 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:35.380268 (+ 5980us) server_negotiation.cc:407] Connection header received
0929 18:32:35.380450 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:35.380456 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:35.380518 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:35.380576 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:35.381393 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.382287 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.383726 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.384067 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.384229 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:35.389550 (+ 5321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:35.389557 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:35.389561 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:35.389621 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:35.391936 (+ 2315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:35.392538 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:35.392541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:35.392543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:35.392631 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:35.392904 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:35.392909 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:35.392911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:35.393146 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:35.393229 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:35.394793 (+ 1564us) server_negotiation.cc:299] Negotiation successful
0929 18:32:35.395097 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 18:32:35.395823 4265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:35.374129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:45571 (local address 127.28.207.66:59999)
0929 18:32:35.376173 (+ 2044us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:35.376226 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:35.377261 (+ 1035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:35.377965 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:35.377976 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:35.378493 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:35.379390 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:35.379413 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.380715 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.380718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:35.382567 (+ 1849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:35.382577 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.383652 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.383659 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:35.383931 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:35.385020 (+ 1089us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:35.385054 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:35.388420 (+ 3366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:35.391062 (+ 2642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:35.391072 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:35.391097 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:35.391612 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:35.391997 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:35.392003 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:35.392008 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:35.392227 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:35.392828 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:35.392840 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:35.393280 (+ 440us) client_negotiation.cc:769] Sending connection context
0929 18:32:35.394281 (+ 1001us) client_negotiation.cc:240] Negotiation successful
0929 18:32:35.394547 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1835,"mutex_wait_us":84,"spinlock_wait_cycles":9472,"thread_start_us":1193,"threads_started":1}
I20260929 18:32:35.395838 4263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:35.374129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:33133 (local address 127.28.207.66:47751)
0929 18:32:35.379718 (+ 5589us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:35.379738 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:35.379878 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:35.380326 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:35.380330 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:35.380603 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:35.380860 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:35.380868 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.382057 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.382061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:35.383106 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:35.383116 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.383859 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.383862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:35.383947 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:35.385020 (+ 1073us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:35.385054 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:35.387906 (+ 2852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:35.391783 (+ 3877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:35.391790 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:35.391795 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:35.392096 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:35.392435 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:35.392441 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:35.392444 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:35.392542 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:35.393098 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:35.393101 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:35.393291 (+ 190us) client_negotiation.cc:769] Sending connection context
0929 18:32:35.394976 (+ 1685us) client_negotiation.cc:240] Negotiation successful
0929 18:32:35.395237 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5450,"mutex_wait_us":99,"spinlock_wait_cycles":54400,"thread_start_us":247,"threads_started":1}
I20260929 18:32:35.397219 4264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:35.374129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.125:32877 (local address 127.28.207.66:58881)
0929 18:32:35.380176 (+ 6047us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:35.380193 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:35.380341 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:35.380752 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:35.380758 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:35.381050 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:35.381248 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:35.381259 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.382477 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.382483 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:35.383576 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:35.383588 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.384757 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.384762 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:35.384915 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:35.385445 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:35.385468 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:35.389294 (+ 3826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:35.392067 (+ 2773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:35.392070 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:35.392072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:35.392371 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:35.392729 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:35.392732 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:35.392733 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:35.392807 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:35.393269 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:35.393272 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:35.393353 (+ 81us) client_negotiation.cc:769] Sending connection context
0929 18:32:35.393571 (+ 218us) client_negotiation.cc:240] Negotiation successful
0929 18:32:35.394065 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5907,"mutex_wait_us":164,"spinlock_wait_cycles":2432,"thread_start_us":210,"threads_started":1}
I20260929 18:32:35.397825 4259 heartbeater.cc:347] Connected to a master server at 127.28.207.124:45571
I20260929 18:32:35.397868 4260 heartbeater.cc:347] Connected to a master server at 127.28.207.126:33133
I20260929 18:32:35.398137 4259 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:35.398136 4260 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:35.398425 4261 heartbeater.cc:347] Connected to a master server at 127.28.207.125:32877
I20260929 18:32:35.398527 4261 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:35.399164 4260 heartbeater.cc:511] Master 127.28.207.126:33133 requested a full tablet report, sending...
I20260929 18:32:35.399283 4261 heartbeater.cc:511] Master 127.28.207.125:32877 requested a full tablet report, sending...
I20260929 18:32:35.399164 4259 heartbeater.cc:511] Master 127.28.207.124:45571 requested a full tablet report, sending...
I20260929 18:32:35.400579 3151 ts_manager.cc:194] Re-registered known tserver with Master: af2018baa42e42b9b287a44255461376 (127.28.207.66:44677)
I20260929 18:32:35.400856 3746 ts_manager.cc:194] Registered new tserver with Master: af2018baa42e42b9b287a44255461376 (127.28.207.66:44677)
I20260929 18:32:35.401417 3151 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.207.66@KRBTEST.COM'} at 127.28.207.66:47751
I20260929 18:32:35.401496 3951 ts_manager.cc:194] Registered new tserver with Master: af2018baa42e42b9b287a44255461376 (127.28.207.66:44677)
W20260929 18:32:35.509733 4266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:35.510082 4266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:35.510159 4266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:35.515545 4266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:35.515632 4266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:35.515717 4266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.67
I20260929 18:32:35.516840 4129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:35.512030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53558 (local address 127.28.207.124:45571)
0929 18:32:35.512217 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:35.512223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:35.512435 (+ 212us) server_negotiation.cc:407] Connection header received
0929 18:32:35.512618 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:35.512621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:35.512694 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:35.512839 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:35.513261 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.514101 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.515527 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.516209 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.516354 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:35.516494 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:35.516640 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 18:32:35.516711 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 18:32:35.516853 4271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:35.511952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:45571 (local address 127.0.0.1:53558)
0929 18:32:35.512312 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:35.512329 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:35.512482 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:35.512907 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:35.512910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:35.512929 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:35.513059 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:35.513066 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.514263 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.514275 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:35.515382 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:35.515393 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.516389 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.516392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:35.516492 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:35.516522 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 18:32:35.516604 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 18:32:35.516688 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":111,"threads_started":1}
I20260929 18:32:35.517951 3961 raft_consensus.cc:1203] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260929 18:32:35.518321 3997 sys_catalog.cc:455] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1250cda9f69445ecadffa350f7d1790c. Latest consensus state: current_term: 1 leader_uuid: "1250cda9f69445ecadffa350f7d1790c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: false } } peers { permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: NON_VOTER last_known_addr { host: "127.28.207.124" port: 45571 } attrs { promote: true } } }
I20260929 18:32:35.518543 3997 sys_catalog.cc:458] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [sys.catalog]: This master's current role is: LEARNER
I20260929 18:32:35.522328 4266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:42331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.67.keytab
--principal=oryx/127.28.207.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.67:36657
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.207.67
--webserver_port=35729
--tserver_master_addrs=127.28.207.126:33133,127.28.207.125:32877,127.28.207.124:45571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.67
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:35.523859 4266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:35.524946 4266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:35.535933 4273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:35.535902 4274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:35.536156 4276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:35.535990 4266 server_base.cc:1034] running on GCE node
I20260929 18:32:35.536628 4266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:35.537407 4266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:35.538575 4266 hybrid_clock.cc:648] HybridClock initialized: now 1790706755538530 us; error 77 us; skew 500 ppm
Sep 29 18:32:35 dist-test-slave-qcpv krb5kdc[3106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706755, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.207.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:35.541781 4266 init.cc:377] Logged in from keytab as oryx/127.28.207.67@KRBTEST.COM (short username oryx)
I20260929 18:32:35.542779 4266 webserver.cc:460] Webserver started at http://127.28.207.67:35729/ using document root <none> and password file <none>
I20260929 18:32:35.543431 4266 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:35.543540 4266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:35.548064 4266 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:32:35.550974 4283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:35.552724 4266 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:32:35.552881 4266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-2/wal
uuid: "37153dd37c404c03a93687dad1b4cdca"
format_stamp: "Formatted at 2026-09-29 18:32:34 on dist-test-slave-qcpv"
server_key: "20eac7ce9446e3012cab8ef9941bde39"
server_key_iv: "e77820779dedf9999f590b80ea482052"
server_key_version: "encryptionkey@0"
I20260929 18:32:35.553496 4266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:35.583570 4266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:35.587725 4266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:35.587940 4266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:35.588698 4266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:32:35.589711 4266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:35.589783 4266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:35.589848 4266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:35.589874 4266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:35.600654 4266 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.67:36657
I20260929 18:32:35.600708 4396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.67:36657 every 8 connection(s)
I20260929 18:32:35.601783 4266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
I20260929 18:32:35.603615 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 4266
Sep 29 18:32:35 dist-test-slave-qcpv krb5kdc[3106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706755, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.207.67@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM
I20260929 18:32:35.619921 4412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:35.607102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53568 (local address 127.28.207.124:45571)
0929 18:32:35.608393 (+ 1291us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:35.608400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:35.608417 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:32:35.608463 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:35.608466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:35.608516 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:35.609966 (+ 1450us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:35.610221 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.611064 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.612534 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.613060 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.613174 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:35.615634 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:35.615638 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:35.615642 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:35.615681 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:35.617561 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:35.618369 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:35.618372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:35.618374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:35.618442 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:35.618767 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:35.618770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:35.618772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:35.619014 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:35.619137 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:35.619591 (+ 454us) server_negotiation.cc:299] Negotiation successful
0929 18:32:35.619713 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1205,"thread_start_us":106,"threads_started":1}
I20260929 18:32:35.620292 4409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:35.606286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50008 (local address 127.28.207.125:32877)
0929 18:32:35.606968 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:35.606973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:35.608117 (+ 1144us) server_negotiation.cc:407] Connection header received
0929 18:32:35.608293 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:35.608296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:35.608357 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:35.608433 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:35.609560 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.610325 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.611643 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.611980 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.612056 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:35.614757 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:35.614761 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:35.614764 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:35.614791 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:35.617224 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:35.618351 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:35.618356 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:35.618357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:35.618442 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:35.618767 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:35.618770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:35.618772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:35.619007 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:35.619137 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:35.620009 (+ 872us) server_negotiation.cc:299] Negotiation successful
0929 18:32:35.620141 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":72,"threads_started":1}
Sep 29 18:32:35 dist-test-slave-qcpv krb5kdc[3106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706755, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.207.67@KRBTEST.COM for oryx/127.28.207.124@KRBTEST.COM
Sep 29 18:32:35 dist-test-slave-qcpv krb5kdc[3106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706755, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.207.67@KRBTEST.COM for oryx/127.28.207.125@KRBTEST.COM
I20260929 18:32:35.624641 4411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:35.606566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52272 (local address 127.28.207.126:33133)
0929 18:32:35.607494 (+ 928us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:35.607501 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:35.607515 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:32:35.607568 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:35.607571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:35.607629 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:35.607761 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:35.608837 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.609810 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.611680 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.612104 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.612190 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:35.616895 (+ 4705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:35.616899 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:35.616902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:35.616929 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:35.620821 (+ 3892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:35.622155 (+ 1334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:35.622159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:35.622163 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:35.622268 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:35.623324 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:35.623329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:35.623330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:35.623645 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:35.623815 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:35.624262 (+ 447us) server_negotiation.cc:299] Negotiation successful
0929 18:32:35.624415 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":855,"thread_start_us":82,"threads_started":1}
I20260929 18:32:35.624827 3190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:35.605139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:49105 (local address 127.28.207.126:33133)
0929 18:32:35.605382 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:35.605387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:35.607081 (+ 1694us) server_negotiation.cc:407] Connection header received
0929 18:32:35.608402 (+ 1321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:35.608407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:35.608482 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:35.608586 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:35.610274 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.611084 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.612664 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.612890 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.613018 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:35.617986 (+ 4968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:35.617991 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:35.617993 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:35.618022 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:35.620093 (+ 2071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:35.622072 (+ 1979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:35.622075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:35.622077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:35.622178 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:35.623028 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:35.623034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:35.623039 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:35.623333 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:35.623416 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:35.624492 (+ 1076us) server_negotiation.cc:299] Negotiation successful
0929 18:32:35.624641 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 18:32:35.628242 4406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:35.604773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:33133 (local address 127.28.207.67:49105)
0929 18:32:35.606886 (+ 2113us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:35.606976 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:35.608093 (+ 1117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:35.609065 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:35.609074 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:35.609432 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:35.610112 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:35.610131 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.611232 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.611235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:35.612545 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:35.612557 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.613211 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.613219 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:35.613435 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:35.614153 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:35.614175 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:35.617799 (+ 3624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:35.621447 (+ 3648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:35.621456 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:35.621482 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:35.621947 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:35.622338 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:35.622343 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:35.622346 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:35.622894 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:35.623504 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:35.623515 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:35.624133 (+ 618us) client_negotiation.cc:769] Sending connection context
0929 18:32:35.625069 (+ 936us) client_negotiation.cc:240] Negotiation successful
0929 18:32:35.625524 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1789,"spinlock_wait_cycles":13056,"thread_start_us":723,"threads_started":1}
I20260929 18:32:35.629321 3876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:35.604420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:46691 (local address 127.28.207.125:32877)
0929 18:32:35.604821 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:35.604828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:35.607243 (+ 2415us) server_negotiation.cc:407] Connection header received
0929 18:32:35.613881 (+ 6638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:35.613886 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:35.613958 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:35.614036 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:35.617817 (+ 3781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.618792 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.621764 (+ 2972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.621969 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.622091 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:35.625387 (+ 3296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:35.625391 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:35.625393 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:35.625422 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:35.627251 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:35.627842 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:35.627846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:35.627847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:35.627945 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:35.628227 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:35.628232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:35.628235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:35.628458 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:35.628549 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:35.629006 (+ 457us) server_negotiation.cc:299] Negotiation successful
0929 18:32:35.629141 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 18:32:35.629910 4407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:35.605382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.125:32877 (local address 127.28.207.67:46691)
0929 18:32:35.607137 (+ 1755us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:35.607154 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:35.613661 (+ 6507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:35.614216 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:35.614219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:35.614819 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:35.615022 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:35.615033 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.619112 (+ 4079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.619117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:35.621526 (+ 2409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:35.621539 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.622239 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.622243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:35.622351 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:35.622960 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:35.622989 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:35.624855 (+ 1866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:35.627383 (+ 2528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:35.627387 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:35.627389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:35.627718 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:35.628047 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:35.628050 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:35.628052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:35.628133 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:35.628605 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:35.628608 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:35.628804 (+ 196us) client_negotiation.cc:769] Sending connection context
0929 18:32:35.629531 (+ 727us) client_negotiation.cc:240] Negotiation successful
0929 18:32:35.629697 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1670,"spinlock_wait_cycles":3968,"thread_start_us":163,"threads_started":1}
I20260929 18:32:35.630111 4129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:35.604420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:49701 (local address 127.28.207.124:45571)
0929 18:32:35.604604 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:35.604611 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:35.607770 (+ 3159us) server_negotiation.cc:407] Connection header received
0929 18:32:35.613582 (+ 5812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:35.613587 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:35.614064 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:35.614120 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:35.615623 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.616525 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.620066 (+ 3541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.620454 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.620552 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:35.625389 (+ 4837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:35.625394 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:35.625397 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:35.625433 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:35.627443 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:35.628070 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:35.628074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:35.628076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:35.628178 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:35.628822 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:35.628826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:35.628828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:35.629068 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:35.629157 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:35.629785 (+ 628us) server_negotiation.cc:299] Negotiation successful
0929 18:32:35.629924 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 18:32:35.630157 4398 heartbeater.cc:347] Connected to a master server at 127.28.207.126:33133
I20260929 18:32:35.630460 4398 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:35.630977 4399 heartbeater.cc:347] Connected to a master server at 127.28.207.125:32877
I20260929 18:32:35.631088 4399 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:35.631242 4399 heartbeater.cc:511] Master 127.28.207.125:32877 requested a full tablet report, sending...
I20260929 18:32:35.631214 4398 heartbeater.cc:511] Master 127.28.207.126:33133 requested a full tablet report, sending...
I20260929 18:32:35.628398 3151 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 18:32:35.632288 4405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:35.605482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:45571 (local address 127.28.207.67:49701)
0929 18:32:35.607632 (+ 2150us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:35.607647 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:35.610543 (+ 2896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:35.614251 (+ 3708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:35.614253 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:35.615279 (+ 1026us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:35.615457 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:35.615466 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.616708 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.616711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:35.617663 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:35.617674 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.621715 (+ 4041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.621718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:35.621843 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:35.622448 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:35.622462 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:35.625210 (+ 2748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:35.627594 (+ 2384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:35.627600 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:35.627603 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:35.627912 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:35.628309 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:35.628312 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:35.628314 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:35.628696 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:35.629224 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:35.629228 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:35.629340 (+ 112us) client_negotiation.cc:769] Sending connection context
0929 18:32:35.630676 (+ 1336us) client_negotiation.cc:240] Negotiation successful
0929 18:32:35.630917 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2083,"spinlock_wait_cycles":1408,"thread_start_us":105,"threads_started":1}
I20260929 18:32:35.632695 3746 ts_manager.cc:194] Registered new tserver with Master: 37153dd37c404c03a93687dad1b4cdca (127.28.207.67:36657)
I20260929 18:32:35.632977 3150 ts_manager.cc:194] Re-registered known tserver with Master: 37153dd37c404c03a93687dad1b4cdca (127.28.207.67:36657)
W20260929 18:32:35.633200 3151 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:32:35.633507 4397 heartbeater.cc:347] Connected to a master server at 127.28.207.124:45571
I20260929 18:32:35.633594 4397 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:35.633760 4397 heartbeater.cc:511] Master 127.28.207.124:45571 requested a full tablet report, sending...
I20260929 18:32:35.633805 3150 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.207.67@KRBTEST.COM'} at 127.28.207.67:49105
I20260929 18:32:35.634593 3951 ts_manager.cc:194] Registered new tserver with Master: 37153dd37c404c03a93687dad1b4cdca (127.28.207.67:36657)
I20260929 18:32:35.637044 4267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1250cda9f69445ecadffa350f7d1790c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: false } } peers { permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: NON_VOTER last_known_addr { host: "127.28.207.124" port: 45571 } attrs { promote: true } } }
I20260929 18:32:35.637188 4267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:35.639389 4267 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c: attempting to promote NON_VOTER e65dbad386c04dd381b04723f4a864e3 to VOTER
I20260929 18:32:35.639801 4267 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: false } } peers { permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 45571 } attrs { promote: false } }
I20260929 18:32:35.641208 3961 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [term 1 LEARNER]: Refusing update from remote peer 1250cda9f69445ecadffa350f7d1790c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260929 18:32:35.641256 3764 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [term 1 FOLLOWER]: Refusing update from remote peer 1250cda9f69445ecadffa350f7d1790c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260929 18:32:35.641635 4415 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 45571 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 18:32:35.641908 4267 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 18:32:35.644618 4417 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, e65dbad386c04dd381b04723f4a864e3 (127.28.207.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: false } } peers { permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 45571 } attrs { promote: false } } }
I20260929 18:32:35.645236 3961 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, e65dbad386c04dd381b04723f4a864e3 (127.28.207.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: false } } peers { permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 45571 } attrs { promote: false } } }
I20260929 18:32:35.646925 3997 sys_catalog.cc:455] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1250cda9f69445ecadffa350f7d1790c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: false } } peers { permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 45571 } attrs { promote: false } } }
I20260929 18:32:35.647115 3997 sys_catalog.cc:458] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:32:35.647804 3764 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, e65dbad386c04dd381b04723f4a864e3 (127.28.207.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: false } } peers { permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 45571 } attrs { promote: false } } }
I20260929 18:32:35.649887 4425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1250cda9f69445ecadffa350f7d1790c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: false } } peers { permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 45571 } attrs { promote: false } } }
I20260929 18:32:35.650041 4425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:32:35.651275 4415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1250cda9f69445ecadffa350f7d1790c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: false } } peers { permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 45571 } attrs { promote: false } } }
I20260929 18:32:35.651496 4415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:35.654323 4416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1250cda9f69445ecadffa350f7d1790c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: false } } peers { permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 45571 } attrs { promote: false } } }
I20260929 18:32:35.654488 4416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:35.654997 4417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1250cda9f69445ecadffa350f7d1790c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: false } } peers { permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 45571 } attrs { promote: false } } }
I20260929 18:32:35.655174 4417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:35.669019 4271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:35.661559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:36657 (local address 127.0.0.1:36506)
0929 18:32:35.661684 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:35.661718 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:35.661962 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:35.662920 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:35.662927 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:35.662955 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:35.663098 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:35.663105 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.664914 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.664917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:35.666155 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:35.666162 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.668500 (+ 2338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.668504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:35.668631 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:35.668654 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:32:35.668734 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 18:32:35.668841 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":30}
I20260929 18:32:35.669404 4426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:35.661771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36506 (local address 127.28.207.67:36657)
0929 18:32:35.662276 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:35.662288 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:35.662315 (+ 27us) server_negotiation.cc:407] Connection header received
0929 18:32:35.662404 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:35.662410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:35.662683 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:35.662941 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:35.663294 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.664782 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.667388 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.668346 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.668573 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:35.668776 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:35.668837 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 18:32:35.668952 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":146,"threads_started":1}
I20260929 18:32:35.669768 4429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:35.662221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:44677 (local address 127.0.0.1:57458)
0929 18:32:35.663877 (+ 1656us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:35.663894 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:35.664005 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:35.664491 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:35.664495 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:35.664513 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:35.664608 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:35.664614 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.666082 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.666085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:35.667438 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:35.667447 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.668500 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.668504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:35.669436 (+ 932us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:35.669467 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 18:32:35.669538 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 18:32:35.669615 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1582,"mutex_wait_us":24,"thread_start_us":170,"threads_started":1}
I20260929 18:32:35.670472 4428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:35.662764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57458 (local address 127.28.207.66:44677)
0929 18:32:35.663976 (+ 1212us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:35.663992 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:35.664022 (+ 30us) server_negotiation.cc:407] Connection header received
0929 18:32:35.664116 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:35.664121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:35.664328 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:35.664502 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:35.664747 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.665946 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.667593 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.668346 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.670010 (+ 1664us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:35.670170 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:35.670219 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 18:32:35.670324 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1088,"thread_start_us":204,"threads_started":1}
I20260929 18:32:35.670661 4427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:35.663333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:37323 (local address 127.0.0.1:53284)
0929 18:32:35.664791 (+ 1458us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:35.664809 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:35.664909 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:35.665567 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:35.665571 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:35.665588 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:35.665708 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:35.665714 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.667285 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.667289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:35.668714 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:35.668724 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.670169 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.670173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:35.670279 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:35.670304 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 18:32:35.670380 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 18:32:35.670470 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1379,"thread_start_us":130,"threads_started":1}
I20260929 18:32:35.671255 4430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:35.663504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53284 (local address 127.28.207.65:37323)
0929 18:32:35.664226 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:35.664234 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:35.665021 (+ 787us) server_negotiation.cc:407] Connection header received
0929 18:32:35.665111 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:35.665116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:35.665295 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:35.665419 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:35.665827 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.667144 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.668886 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.669818 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.669919 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:35.670082 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:35.670985 (+ 903us) server_negotiation.cc:299] Negotiation successful
0929 18:32:35.671102 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":97,"threads_started":1}
I20260929 18:32:35.673708 4331 tablet_service.cc:1467] Processing CreateTablet for tablet 310f209325f04859b20911f5259177ab (DEFAULT_TABLE table=test-table [id=e029dc5fa8784a48b3dd4989c278181d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:35.673712 4191 tablet_service.cc:1467] Processing CreateTablet for tablet 310f209325f04859b20911f5259177ab (DEFAULT_TABLE table=test-table [id=e029dc5fa8784a48b3dd4989c278181d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:35.675345 4331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 310f209325f04859b20911f5259177ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:35.675371 4191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 310f209325f04859b20911f5259177ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:35.675748 4056 tablet_service.cc:1467] Processing CreateTablet for tablet 310f209325f04859b20911f5259177ab (DEFAULT_TABLE table=test-table [id=e029dc5fa8784a48b3dd4989c278181d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:35.676985 4056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 310f209325f04859b20911f5259177ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:35.684867 4431 tablet_bootstrap.cc:492] T 310f209325f04859b20911f5259177ab P 52f601bc57eb4420854cb9073fdd9caa: Bootstrap starting.
I20260929 18:32:35.687644 4431 tablet_bootstrap.cc:654] T 310f209325f04859b20911f5259177ab P 52f601bc57eb4420854cb9073fdd9caa: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:35.688153 4433 tablet_bootstrap.cc:492] T 310f209325f04859b20911f5259177ab P af2018baa42e42b9b287a44255461376: Bootstrap starting.
I20260929 18:32:35.688577 4431 log.cc:824] T 310f209325f04859b20911f5259177ab P 52f601bc57eb4420854cb9073fdd9caa: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:35.689491 4432 tablet_bootstrap.cc:492] T 310f209325f04859b20911f5259177ab P 37153dd37c404c03a93687dad1b4cdca: Bootstrap starting.
I20260929 18:32:35.691301 4433 tablet_bootstrap.cc:654] T 310f209325f04859b20911f5259177ab P af2018baa42e42b9b287a44255461376: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:35.691690 4432 tablet_bootstrap.cc:654] T 310f209325f04859b20911f5259177ab P 37153dd37c404c03a93687dad1b4cdca: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:35.691792 4431 tablet_bootstrap.cc:492] T 310f209325f04859b20911f5259177ab P 52f601bc57eb4420854cb9073fdd9caa: No bootstrap required, opened a new log
I20260929 18:32:35.692076 4431 ts_tablet_manager.cc:1397] T 310f209325f04859b20911f5259177ab P 52f601bc57eb4420854cb9073fdd9caa: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 18:32:35.692478 4433 log.cc:824] T 310f209325f04859b20911f5259177ab P af2018baa42e42b9b287a44255461376: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:35.692600 4432 log.cc:824] T 310f209325f04859b20911f5259177ab P 37153dd37c404c03a93687dad1b4cdca: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:35.695194 4432 tablet_bootstrap.cc:492] T 310f209325f04859b20911f5259177ab P 37153dd37c404c03a93687dad1b4cdca: No bootstrap required, opened a new log
I20260929 18:32:35.695195 4433 tablet_bootstrap.cc:492] T 310f209325f04859b20911f5259177ab P af2018baa42e42b9b287a44255461376: No bootstrap required, opened a new log
I20260929 18:32:35.695410 4433 ts_tablet_manager.cc:1397] T 310f209325f04859b20911f5259177ab P af2018baa42e42b9b287a44255461376: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260929 18:32:35.695410 4432 ts_tablet_manager.cc:1397] T 310f209325f04859b20911f5259177ab P 37153dd37c404c03a93687dad1b4cdca: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 18:32:35.696977 4431 raft_consensus.cc:348] T 310f209325f04859b20911f5259177ab P 52f601bc57eb4420854cb9073fdd9caa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f601bc57eb4420854cb9073fdd9caa" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 37323 } } peers { permanent_uuid: "37153dd37c404c03a93687dad1b4cdca" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 36657 } } peers { permanent_uuid: "af2018baa42e42b9b287a44255461376" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 44677 } }
I20260929 18:32:35.697372 4431 raft_consensus.cc:374] T 310f209325f04859b20911f5259177ab P 52f601bc57eb4420854cb9073fdd9caa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:35.697414 4431 raft_consensus.cc:729] T 310f209325f04859b20911f5259177ab P 52f601bc57eb4420854cb9073fdd9caa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f601bc57eb4420854cb9073fdd9caa, State: Initialized, Role: FOLLOWER
I20260929 18:32:35.699599 4431 consensus_queue.cc:260] T 310f209325f04859b20911f5259177ab P 52f601bc57eb4420854cb9073fdd9caa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f601bc57eb4420854cb9073fdd9caa" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 37323 } } peers { permanent_uuid: "37153dd37c404c03a93687dad1b4cdca" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 36657 } } peers { permanent_uuid: "af2018baa42e42b9b287a44255461376" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 44677 } }
I20260929 18:32:35.700642 4433 raft_consensus.cc:348] T 310f209325f04859b20911f5259177ab P af2018baa42e42b9b287a44255461376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f601bc57eb4420854cb9073fdd9caa" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 37323 } } peers { permanent_uuid: "37153dd37c404c03a93687dad1b4cdca" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 36657 } } peers { permanent_uuid: "af2018baa42e42b9b287a44255461376" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 44677 } }
I20260929 18:32:35.700999 4433 raft_consensus.cc:374] T 310f209325f04859b20911f5259177ab P af2018baa42e42b9b287a44255461376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:35.701072 4433 raft_consensus.cc:729] T 310f209325f04859b20911f5259177ab P af2018baa42e42b9b287a44255461376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af2018baa42e42b9b287a44255461376, State: Initialized, Role: FOLLOWER
I20260929 18:32:35.701109 4431 ts_tablet_manager.cc:1428] T 310f209325f04859b20911f5259177ab P 52f601bc57eb4420854cb9073fdd9caa: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260929 18:32:35.701321 4123 heartbeater.cc:503] Master 127.28.207.126:33133 was elected leader, sending a full tablet report...
I20260929 18:32:35.701582 4433 consensus_queue.cc:260] T 310f209325f04859b20911f5259177ab P af2018baa42e42b9b287a44255461376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f601bc57eb4420854cb9073fdd9caa" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 37323 } } peers { permanent_uuid: "37153dd37c404c03a93687dad1b4cdca" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 36657 } } peers { permanent_uuid: "af2018baa42e42b9b287a44255461376" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 44677 } }
I20260929 18:32:35.701633 4432 raft_consensus.cc:348] T 310f209325f04859b20911f5259177ab P 37153dd37c404c03a93687dad1b4cdca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f601bc57eb4420854cb9073fdd9caa" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 37323 } } peers { permanent_uuid: "37153dd37c404c03a93687dad1b4cdca" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 36657 } } peers { permanent_uuid: "af2018baa42e42b9b287a44255461376" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 44677 } }
I20260929 18:32:35.701875 4432 raft_consensus.cc:374] T 310f209325f04859b20911f5259177ab P 37153dd37c404c03a93687dad1b4cdca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:35.701930 4432 raft_consensus.cc:729] T 310f209325f04859b20911f5259177ab P 37153dd37c404c03a93687dad1b4cdca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37153dd37c404c03a93687dad1b4cdca, State: Initialized, Role: FOLLOWER
I20260929 18:32:35.702328 4432 consensus_queue.cc:260] T 310f209325f04859b20911f5259177ab P 37153dd37c404c03a93687dad1b4cdca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f601bc57eb4420854cb9073fdd9caa" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 37323 } } peers { permanent_uuid: "37153dd37c404c03a93687dad1b4cdca" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 36657 } } peers { permanent_uuid: "af2018baa42e42b9b287a44255461376" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 44677 } }
I20260929 18:32:35.702801 4433 ts_tablet_manager.cc:1428] T 310f209325f04859b20911f5259177ab P af2018baa42e42b9b287a44255461376: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 18:32:35.703320 4432 ts_tablet_manager.cc:1428] T 310f209325f04859b20911f5259177ab P 37153dd37c404c03a93687dad1b4cdca: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260929 18:32:35.703773 4260 heartbeater.cc:503] Master 127.28.207.126:33133 was elected leader, sending a full tablet report...
I20260929 18:32:35.703881 4398 heartbeater.cc:503] Master 127.28.207.126:33133 was elected leader, sending a full tablet report...
W20260929 18:32:35.853585 4400 tablet.cc:2369] T 310f209325f04859b20911f5259177ab P 37153dd37c404c03a93687dad1b4cdca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:32:35.856155 4438 raft_consensus.cc:482] T 310f209325f04859b20911f5259177ab P af2018baa42e42b9b287a44255461376 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:32:35.856513 4438 raft_consensus.cc:504] T 310f209325f04859b20911f5259177ab P af2018baa42e42b9b287a44255461376 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f601bc57eb4420854cb9073fdd9caa" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 37323 } } peers { permanent_uuid: "37153dd37c404c03a93687dad1b4cdca" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 36657 } } peers { permanent_uuid: "af2018baa42e42b9b287a44255461376" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 44677 } }
I20260929 18:32:35.858186 4438 leader_election.cc:288] T 310f209325f04859b20911f5259177ab P af2018baa42e42b9b287a44255461376 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52f601bc57eb4420854cb9073fdd9caa (127.28.207.65:37323), 37153dd37c404c03a93687dad1b4cdca (127.28.207.67:36657)
I20260929 18:32:35.863142 4430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:35.858911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:44565 (local address 127.28.207.65:37323)
0929 18:32:35.859059 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:35.859063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:35.859079 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:32:35.859144 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:35.859146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:35.859237 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:35.859368 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:35.859802 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.860475 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.861841 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.862378 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.862503 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:35.862560 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:35.862904 (+ 344us) server_negotiation.cc:299] Negotiation successful
0929 18:32:35.862995 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 18:32:35.863180 4264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:35.858583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:37323 (local address 127.28.207.66:44565)
0929 18:32:35.858721 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:35.858844 (+ 123us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:35.859023 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:35.859437 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:35.859441 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:35.859465 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:35.859614 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:35.859625 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.860624 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.860627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:35.861705 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:35.861715 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.862530 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.862534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:35.862687 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:35.862724 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 18:32:35.862824 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 18:32:35.862963 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 18:32:35.863947 4075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "310f209325f04859b20911f5259177ab" candidate_uuid: "af2018baa42e42b9b287a44255461376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f601bc57eb4420854cb9073fdd9caa" is_pre_election: true
I20260929 18:32:35.864024 4263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:35.858584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:36657 (local address 127.28.207.66:33217)
0929 18:32:35.859285 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:35.859305 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:35.859394 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:35.859794 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:35.859799 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:35.859821 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:35.859933 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:35.859939 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.860853 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.860856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:35.862282 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:35.862297 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.863562 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.863567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:35.863672 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:35.863691 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 18:32:35.863798 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 18:32:35.863887 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"mutex_wait_us":524}
I20260929 18:32:35.864045 4426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:35.858698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:33217 (local address 127.28.207.67:36657)
0929 18:32:35.858843 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:35.858849 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:35.859448 (+ 599us) server_negotiation.cc:407] Connection header received
0929 18:32:35.859512 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:35.859515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:35.859568 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:35.859672 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:35.860097 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.860680 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.862494 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.863273 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.863423 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:35.863507 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:35.863819 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 18:32:35.863913 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:32:35.864355 4075 raft_consensus.cc:2454] T 310f209325f04859b20911f5259177ab P 52f601bc57eb4420854cb9073fdd9caa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af2018baa42e42b9b287a44255461376 in term 0.
I20260929 18:32:35.864866 4351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "310f209325f04859b20911f5259177ab" candidate_uuid: "af2018baa42e42b9b287a44255461376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37153dd37c404c03a93687dad1b4cdca" is_pre_election: true
I20260929 18:32:35.864976 4147 leader_election.cc:302] T 310f209325f04859b20911f5259177ab P af2018baa42e42b9b287a44255461376 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52f601bc57eb4420854cb9073fdd9caa, af2018baa42e42b9b287a44255461376; no voters:
I20260929 18:32:35.865160 4351 raft_consensus.cc:2454] T 310f209325f04859b20911f5259177ab P 37153dd37c404c03a93687dad1b4cdca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af2018baa42e42b9b287a44255461376 in term 0.
I20260929 18:32:35.865461 4438 raft_consensus.cc:2781] T 310f209325f04859b20911f5259177ab P af2018baa42e42b9b287a44255461376 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:32:35.865598 4438 raft_consensus.cc:482] T 310f209325f04859b20911f5259177ab P af2018baa42e42b9b287a44255461376 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:32:35.865661 4438 raft_consensus.cc:3037] T 310f209325f04859b20911f5259177ab P af2018baa42e42b9b287a44255461376 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:35.866909 4438 raft_consensus.cc:504] T 310f209325f04859b20911f5259177ab P af2018baa42e42b9b287a44255461376 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f601bc57eb4420854cb9073fdd9caa" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 37323 } } peers { permanent_uuid: "37153dd37c404c03a93687dad1b4cdca" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 36657 } } peers { permanent_uuid: "af2018baa42e42b9b287a44255461376" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 44677 } }
I20260929 18:32:35.867352 4438 leader_election.cc:288] T 310f209325f04859b20911f5259177ab P af2018baa42e42b9b287a44255461376 [CANDIDATE]: Term 1 election: Requested vote from peers 52f601bc57eb4420854cb9073fdd9caa (127.28.207.65:37323), 37153dd37c404c03a93687dad1b4cdca (127.28.207.67:36657)
I20260929 18:32:35.868291 4351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "310f209325f04859b20911f5259177ab" candidate_uuid: "af2018baa42e42b9b287a44255461376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37153dd37c404c03a93687dad1b4cdca"
I20260929 18:32:35.868299 4075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "310f209325f04859b20911f5259177ab" candidate_uuid: "af2018baa42e42b9b287a44255461376" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f601bc57eb4420854cb9073fdd9caa"
I20260929 18:32:35.868599 4075 raft_consensus.cc:3037] T 310f209325f04859b20911f5259177ab P 52f601bc57eb4420854cb9073fdd9caa [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:35.868599 4351 raft_consensus.cc:3037] T 310f209325f04859b20911f5259177ab P 37153dd37c404c03a93687dad1b4cdca [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:35.870509 4351 raft_consensus.cc:2454] T 310f209325f04859b20911f5259177ab P 37153dd37c404c03a93687dad1b4cdca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af2018baa42e42b9b287a44255461376 in term 1.
I20260929 18:32:35.870509 4075 raft_consensus.cc:2454] T 310f209325f04859b20911f5259177ab P 52f601bc57eb4420854cb9073fdd9caa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af2018baa42e42b9b287a44255461376 in term 1.
I20260929 18:32:35.871270 4147 leader_election.cc:302] T 310f209325f04859b20911f5259177ab P af2018baa42e42b9b287a44255461376 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 37153dd37c404c03a93687dad1b4cdca, 52f601bc57eb4420854cb9073fdd9caa, af2018baa42e42b9b287a44255461376; no voters:
I20260929 18:32:35.871575 4438 raft_consensus.cc:2781] T 310f209325f04859b20911f5259177ab P af2018baa42e42b9b287a44255461376 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:32:35.871841 4438 raft_consensus.cc:686] T 310f209325f04859b20911f5259177ab P af2018baa42e42b9b287a44255461376 [term 1 LEADER]: Becoming Leader. State: Replica: af2018baa42e42b9b287a44255461376, State: Running, Role: LEADER
I20260929 18:32:35.872530 4438 consensus_queue.cc:237] T 310f209325f04859b20911f5259177ab P af2018baa42e42b9b287a44255461376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f601bc57eb4420854cb9073fdd9caa" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 37323 } } peers { permanent_uuid: "37153dd37c404c03a93687dad1b4cdca" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 36657 } } peers { permanent_uuid: "af2018baa42e42b9b287a44255461376" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 44677 } }
W20260929 18:32:35.873101 4262 tablet.cc:2369] T 310f209325f04859b20911f5259177ab P af2018baa42e42b9b287a44255461376: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:32:35.877266 3150 catalog_manager.cc:5495] T 310f209325f04859b20911f5259177ab P af2018baa42e42b9b287a44255461376 reported cstate change: term changed from 0 to 1, leader changed from <none> to af2018baa42e42b9b287a44255461376 (127.28.207.66). New cstate: current_term: 1 leader_uuid: "af2018baa42e42b9b287a44255461376" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f601bc57eb4420854cb9073fdd9caa" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 37323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37153dd37c404c03a93687dad1b4cdca" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 36657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af2018baa42e42b9b287a44255461376" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 44677 } health_report { overall_health: HEALTHY } } }
W20260929 18:32:35.885552 4125 tablet.cc:2369] T 310f209325f04859b20911f5259177ab P 52f601bc57eb4420854cb9073fdd9caa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:32:35.918375 4428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:35.914040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57470 (local address 127.28.207.66:44677)
0929 18:32:35.914231 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:35.914236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:35.914249 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:32:35.914326 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:35.914328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:35.914409 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:35.914539 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:32:35.914812 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.915397 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.916650 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.916857 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.916957 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:35.917287 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:32:35.917761 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:32:35.917858 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:35.918063 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 18:32:35.918131 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 18:32:35.926582 4351 raft_consensus.cc:1261] T 310f209325f04859b20911f5259177ab P 37153dd37c404c03a93687dad1b4cdca [term 1 FOLLOWER]: Refusing update from remote peer af2018baa42e42b9b287a44255461376: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:32:35.926702 4075 raft_consensus.cc:1261] T 310f209325f04859b20911f5259177ab P 52f601bc57eb4420854cb9073fdd9caa [term 1 FOLLOWER]: Refusing update from remote peer af2018baa42e42b9b287a44255461376: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:32:35.927650 4438 consensus_queue.cc:1035] T 310f209325f04859b20911f5259177ab P af2018baa42e42b9b287a44255461376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37153dd37c404c03a93687dad1b4cdca" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 36657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:32:35.927950 4440 consensus_queue.cc:1035] T 310f209325f04859b20911f5259177ab P af2018baa42e42b9b287a44255461376 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f601bc57eb4420854cb9073fdd9caa" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 37323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:32:35.945997 4444 mvcc.cc:205] Tried to move back new op lower bound from 7334734872264826880 to 7334734872058605568. Current Snapshot: MvccSnapshot[applied={T|T < 7334734872264826880}]
I20260929 18:32:35.954015 3150 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:32:35.954195 3150 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:32:35.959846 3150 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1250cda9f69445ecadffa350f7d1790c: Sending DeleteTablet for 3 replicas of tablet 310f209325f04859b20911f5259177ab
I20260929 18:32:35.960920 4331 tablet_service.cc:1514] Processing DeleteTablet for tablet 310f209325f04859b20911f5259177ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:35 UTC) from {username='oryx', principal='oryx/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:36506
I20260929 18:32:35.961113 4056 tablet_service.cc:1514] Processing DeleteTablet for tablet 310f209325f04859b20911f5259177ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:35 UTC) from {username='oryx', principal='oryx/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:53284
I20260929 18:32:35.961575 4191 tablet_service.cc:1514] Processing DeleteTablet for tablet 310f209325f04859b20911f5259177ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:35 UTC) from {username='oryx', principal='oryx/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:57458
I20260929 18:32:35.962009 4455 tablet_replica.cc:330] T 310f209325f04859b20911f5259177ab P af2018baa42e42b9b287a44255461376: stopping tablet replica
I20260929 18:32:35.962395 4455 raft_consensus.cc:2229] T 310f209325f04859b20911f5259177ab P af2018baa42e42b9b287a44255461376 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:32:35.962505 4454 tablet_replica.cc:330] T 310f209325f04859b20911f5259177ab P 52f601bc57eb4420854cb9073fdd9caa: stopping tablet replica
I20260929 18:32:35.962762 4454 raft_consensus.cc:2229] T 310f209325f04859b20911f5259177ab P 52f601bc57eb4420854cb9073fdd9caa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:32:35.962906 4455 raft_consensus.cc:2258] T 310f209325f04859b20911f5259177ab P af2018baa42e42b9b287a44255461376 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:35.963093 4454 raft_consensus.cc:2258] T 310f209325f04859b20911f5259177ab P 52f601bc57eb4420854cb9073fdd9caa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:35.963575 4453 tablet_replica.cc:330] T 310f209325f04859b20911f5259177ab P 37153dd37c404c03a93687dad1b4cdca: stopping tablet replica
I20260929 18:32:35.963848 4453 raft_consensus.cc:2229] T 310f209325f04859b20911f5259177ab P 37153dd37c404c03a93687dad1b4cdca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:32:35.964144 4453 raft_consensus.cc:2258] T 310f209325f04859b20911f5259177ab P 37153dd37c404c03a93687dad1b4cdca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:35.965627 4454 ts_tablet_manager.cc:1905] T 310f209325f04859b20911f5259177ab P 52f601bc57eb4420854cb9073fdd9caa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:32:35.965627 4455 ts_tablet_manager.cc:1905] T 310f209325f04859b20911f5259177ab P af2018baa42e42b9b287a44255461376: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:32:35.966238 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 3119
I20260929 18:32:35.969106 4454 ts_tablet_manager.cc:1918] T 310f209325f04859b20911f5259177ab P 52f601bc57eb4420854cb9073fdd9caa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:32:35.969216 4454 log.cc:1196] T 310f209325f04859b20911f5259177ab P 52f601bc57eb4420854cb9073fdd9caa: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-0/wal/wals/310f209325f04859b20911f5259177ab
I20260929 18:32:35.969583 4454 ts_tablet_manager.cc:1939] T 310f209325f04859b20911f5259177ab P 52f601bc57eb4420854cb9073fdd9caa: Deleting consensus metadata
I20260929 18:32:35.970893 4455 ts_tablet_manager.cc:1918] T 310f209325f04859b20911f5259177ab P af2018baa42e42b9b287a44255461376: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:32:35.971012 4455 log.cc:1196] T 310f209325f04859b20911f5259177ab P af2018baa42e42b9b287a44255461376: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-1/wal/wals/310f209325f04859b20911f5259177ab
I20260929 18:32:35.971321 4455 ts_tablet_manager.cc:1939] T 310f209325f04859b20911f5259177ab P af2018baa42e42b9b287a44255461376: Deleting consensus metadata
I20260929 18:32:35.972389 4453 ts_tablet_manager.cc:1905] T 310f209325f04859b20911f5259177ab P 37153dd37c404c03a93687dad1b4cdca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:32:35.975219 4453 ts_tablet_manager.cc:1918] T 310f209325f04859b20911f5259177ab P 37153dd37c404c03a93687dad1b4cdca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:32:35.975499 4453 log.cc:1196] T 310f209325f04859b20911f5259177ab P 37153dd37c404c03a93687dad1b4cdca: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-2/wal/wals/310f209325f04859b20911f5259177ab
I20260929 18:32:35.975896 4453 ts_tablet_manager.cc:1939] T 310f209325f04859b20911f5259177ab P 37153dd37c404c03a93687dad1b4cdca: Deleting consensus metadata
W20260929 18:32:35.978724 4012 connection.cc:535] server connection from 127.0.0.1:53284 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260929 18:32:35.978789 4149 connection.cc:535] server connection from 127.0.0.1:57458 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260929 18:32:35.979303 4284 connection.cc:535] server connection from 127.0.0.1:36506 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 18:32:35.981124 3876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:35.964537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50024 (local address 127.28.207.125:32877)
0929 18:32:35.964700 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:35.964708 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:35.969341 (+ 4633us) server_negotiation.cc:407] Connection header received
0929 18:32:35.969582 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:35.969586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:35.969647 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:35.969730 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:35.972424 (+ 2694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.973155 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.974424 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.974780 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.974912 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:35.977015 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:35.977019 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:35.977022 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:35.977053 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:35.978718 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:35.979308 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:35.979314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:35.979319 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:35.979406 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:35.979728 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:35.979731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:35.979733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:35.980008 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:35.980133 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:35.980815 (+ 682us) server_negotiation.cc:299] Negotiation successful
0929 18:32:35.980938 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:32:35.992651 4129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:35.964295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53582 (local address 127.28.207.124:45571)
0929 18:32:35.964445 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:35.964450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:35.982046 (+ 17596us) server_negotiation.cc:407] Connection header received
0929 18:32:35.982233 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:35.982236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:35.982300 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:35.982376 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:35.983128 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.983869 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.984977 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.985229 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.985344 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:35.987073 (+ 1729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:35.987076 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:35.987079 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:35.987110 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:35.988942 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:35.989445 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:35.989449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:35.989454 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:35.989855 (+ 401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:35.990518 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:35.990525 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:35.990528 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:35.990837 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:35.991098 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:35.992210 (+ 1112us) server_negotiation.cc:299] Negotiation successful
0929 18:32:35.992349 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:32:35.996546 3876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:35.984064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50030 (local address 127.28.207.125:32877)
0929 18:32:35.984174 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:35.984178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:35.984190 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:32:35.984225 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:35.984227 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:35.984268 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:35.984361 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:35.986237 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.987068 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.988406 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.988789 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.988897 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:35.991475 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:35.991485 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:35.991488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:35.991528 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:35.994013 (+ 2485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:35.994663 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:35.994667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:35.994669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:35.994758 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:35.995137 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:35.995142 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:35.995144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:35.995452 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:35.995616 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:35.996210 (+ 594us) server_negotiation.cc:299] Negotiation successful
0929 18:32:35.996348 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:32:35.999245 4412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:35.984662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53594 (local address 127.28.207.124:45571)
0929 18:32:35.984840 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:35.984846 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:35.987329 (+ 2483us) server_negotiation.cc:407] Connection header received
0929 18:32:35.987585 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:35.987589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:35.987652 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:35.987712 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:35.988957 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.989773 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.991805 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:35.992230 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:35.992322 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:35.994985 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:35.994990 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:35.994993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:35.995020 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:35.996772 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:35.997437 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:35.997441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:35.997444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:35.997552 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:35.997974 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:35.997977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:35.997978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:35.998209 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:35.998328 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:35.998688 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 18:32:35.998818 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:32:36.945171 4464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:36.944450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:33133 (local address 127.28.207.65:59985)
0929 18:32:36.944839 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:36.944968 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:33133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":74,"threads_started":1}
I20260929 18:32:36.945240 4463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:36.944434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:33133 (local address 127.28.207.67:45261)
0929 18:32:36.945045 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:36.945136 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:33133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":91,"threads_started":1}
W20260929 18:32:36.945607 4398 heartbeater.cc:644] Failed to heartbeat to 127.28.207.126:33133 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.207.126:33133: connect: Connection refused (error 111)
W20260929 18:32:36.945637 4123 heartbeater.cc:644] Failed to heartbeat to 127.28.207.126:33133 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.207.126:33133: connect: Connection refused (error 111)
I20260929 18:32:36.947162 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:36.946409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:33133 (local address 127.28.207.66:57757)
0929 18:32:36.946931 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:36.947049 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:33133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":214,"threads_started":1}
W20260929 18:32:36.947566 4260 heartbeater.cc:644] Failed to heartbeat to 127.28.207.126:33133 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.28.207.126:33133: connect: Connection refused (error 111)
I20260929 18:32:37.460456 4466 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1250cda9f69445ecadffa350f7d1790c)
I20260929 18:32:37.460850 4467 raft_consensus.cc:482] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1250cda9f69445ecadffa350f7d1790c)
I20260929 18:32:37.460803 4466 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: false } } peers { permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 45571 } attrs { promote: false } }
I20260929 18:32:37.460995 4467 raft_consensus.cc:504] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: false } } peers { permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 45571 } attrs { promote: false } }
I20260929 18:32:37.461573 4467 leader_election.cc:288] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1250cda9f69445ecadffa350f7d1790c (127.28.207.126:33133), 9f296fd5b96e4975b59022170377b843 (127.28.207.125:32877)
I20260929 18:32:37.462740 4466 leader_election.cc:288] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1250cda9f69445ecadffa350f7d1790c (127.28.207.126:33133), e65dbad386c04dd381b04723f4a864e3 (127.28.207.124:45571)
I20260929 18:32:37.462829 4469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:37.461900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:33133 (local address 127.0.0.1:52682)
0929 18:32:37.462371 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:37.462487 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:33133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":121,"threads_started":1}
W20260929 18:32:37.463052 3932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.207.126:33133: connect: Connection refused (error 111)
I20260929 18:32:37.463212 4471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:37.462401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:33133 (local address 127.0.0.1:52688)
0929 18:32:37.462795 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:37.463047 (+ 252us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:33133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":79,"threads_started":1}
W20260929 18:32:37.463495 3740 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.207.126:33133: connect: Connection refused (error 111)
I20260929 18:32:37.464099 4469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:37.463848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:33133 (local address 127.0.0.1:52690)
0929 18:32:37.463986 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:37.464037 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:33133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260929 18:32:37.464221 3932 leader_election.cc:334] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1250cda9f69445ecadffa350f7d1790c (127.28.207.126:33133): Network error: Client connection negotiation failed: client connection to 127.28.207.126:33133: connect: Connection refused (error 111)
I20260929 18:32:37.464612 4471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:37.464301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:33133 (local address 127.0.0.1:52702)
0929 18:32:37.464497 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:37.464547 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:33133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
W20260929 18:32:37.464735 3740 leader_election.cc:334] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1250cda9f69445ecadffa350f7d1790c (127.28.207.126:33133): Network error: Client connection negotiation failed: client connection to 127.28.207.126:33133: connect: Connection refused (error 111)
I20260929 18:32:37.466682 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:37.461995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50040 (local address 127.28.207.125:32877)
0929 18:32:37.462301 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:37.462309 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:37.462829 (+ 520us) server_negotiation.cc:407] Connection header received
0929 18:32:37.462908 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:37.462912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:37.462958 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:37.463033 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:37.463430 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:37.464172 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:37.465558 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:37.466078 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:37.466210 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:37.466268 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:37.466484 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 18:32:37.466556 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":229,"thread_start_us":98,"threads_started":1}
I20260929 18:32:37.466688 4468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:37.461875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.125:32877 (local address 127.0.0.1:50040)
0929 18:32:37.462206 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:37.462601 (+ 395us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:37.462735 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:37.463147 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:37.463149 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:37.463170 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:37.463302 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:37.463309 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:37.464318 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:37.464323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:37.465421 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:37.465431 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:37.466204 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:37.466208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:37.466361 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:37.466382 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 18:32:37.466469 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 18:32:37.466557 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":48,"thread_start_us":70,"threads_started":1}
I20260929 18:32:37.467136 3764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e65dbad386c04dd381b04723f4a864e3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9f296fd5b96e4975b59022170377b843" is_pre_election: true
I20260929 18:32:37.467193 4472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:37.462523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:45571 (local address 127.0.0.1:53606)
0929 18:32:37.463002 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:37.463018 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:37.463142 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:37.463682 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:37.463685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:37.463703 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:37.463849 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:37.463869 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:37.464905 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:37.464912 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:37.465977 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:37.465986 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:37.466709 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:37.466712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:37.466819 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:37.466896 (+ 77us) client_negotiation.cc:769] Sending connection context
0929 18:32:37.466960 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 18:32:37.467045 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":61,"thread_start_us":96,"threads_started":1}
I20260929 18:32:37.467324 3764 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e65dbad386c04dd381b04723f4a864e3 in term 1.
I20260929 18:32:37.467425 4473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:37.462685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53606 (local address 127.28.207.124:45571)
0929 18:32:37.463339 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:37.463346 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:37.463361 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:32:37.463412 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:37.463414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:37.463466 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:37.463556 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:37.463986 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:37.464767 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:37.466094 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:37.466582 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:37.466716 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:37.467143 (+ 427us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:37.467180 (+ 37us) server_negotiation.cc:299] Negotiation successful
0929 18:32:37.467278 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":571,"thread_start_us":63,"threads_started":1}
I20260929 18:32:37.467646 3931 leader_election.cc:302] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9f296fd5b96e4975b59022170377b843, e65dbad386c04dd381b04723f4a864e3; no voters: 1250cda9f69445ecadffa350f7d1790c
I20260929 18:32:37.467859 3961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9f296fd5b96e4975b59022170377b843" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e65dbad386c04dd381b04723f4a864e3" is_pre_election: true
I20260929 18:32:37.467941 4467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 18:32:37.468009 4467 raft_consensus.cc:482] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1250cda9f69445ecadffa350f7d1790c)
I20260929 18:32:37.468061 4467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [term 1 FOLLOWER]: Advancing to term 2
I20260929 18:32:37.469311 4467 raft_consensus.cc:504] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: false } } peers { permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 45571 } attrs { promote: false } }
I20260929 18:32:37.469616 3961 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f296fd5b96e4975b59022170377b843 in current term 2: Already voted for candidate e65dbad386c04dd381b04723f4a864e3 in this term.
I20260929 18:32:37.469769 4467 leader_election.cc:288] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [CANDIDATE]: Term 2 election: Requested vote from peers 1250cda9f69445ecadffa350f7d1790c (127.28.207.126:33133), 9f296fd5b96e4975b59022170377b843 (127.28.207.125:32877)
I20260929 18:32:37.470083 4468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:37.469877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:33133 (local address 127.0.0.1:52714)
0929 18:32:37.469969 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:37.470022 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:33133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260929 18:32:37.470147 3764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e65dbad386c04dd381b04723f4a864e3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9f296fd5b96e4975b59022170377b843"
I20260929 18:32:37.470331 3764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [term 1 FOLLOWER]: Advancing to term 2
I20260929 18:32:37.470480 3738 leader_election.cc:302] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f296fd5b96e4975b59022170377b843; no voters: 1250cda9f69445ecadffa350f7d1790c, e65dbad386c04dd381b04723f4a864e3
I20260929 18:32:37.471021 4468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:37.470682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:33133 (local address 127.0.0.1:52716)
0929 18:32:37.470897 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:37.470953 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:33133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260929 18:32:37.471207 3932 leader_election.cc:334] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1250cda9f69445ecadffa350f7d1790c (127.28.207.126:33133): Network error: Client connection negotiation failed: client connection to 127.28.207.126:33133: connect: Connection refused (error 111)
I20260929 18:32:37.471747 3764 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e65dbad386c04dd381b04723f4a864e3 in term 2.
I20260929 18:32:37.472021 4466 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:32:37.472163 3931 leader_election.cc:302] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9f296fd5b96e4975b59022170377b843, e65dbad386c04dd381b04723f4a864e3; no voters: 1250cda9f69445ecadffa350f7d1790c
I20260929 18:32:37.472369 4467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 18:32:37.472589 4467 raft_consensus.cc:686] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [term 2 LEADER]: Becoming Leader. State: Replica: e65dbad386c04dd381b04723f4a864e3, State: Running, Role: LEADER
I20260929 18:32:37.472868 4467 consensus_queue.cc:237] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: false } } peers { permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 45571 } attrs { promote: false } }
I20260929 18:32:37.474105 4476 sys_catalog.cc:455] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e65dbad386c04dd381b04723f4a864e3. Latest consensus state: current_term: 2 leader_uuid: "e65dbad386c04dd381b04723f4a864e3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: false } } peers { permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 45571 } attrs { promote: false } } }
I20260929 18:32:37.474282 4476 sys_catalog.cc:458] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:37.474658 4478 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:32:37.476969 4478 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:32:37.477377 4478 catalog_manager.cc:1259] Loaded cluster ID: 41ad7488d0df4de6922e7f2e0903f63a
I20260929 18:32:37.477427 4478 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:32:37.477984 4478 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:32:37.478379 4478 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3: Loaded TSK: 0
I20260929 18:32:37.479120 4478 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:32:37.491442 3951 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53594:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:32:37.494809 3951 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:32:37.498383 4468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:37.498089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:33133 (local address 127.0.0.1:52724)
0929 18:32:37.498224 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:37.498292 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:33133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 18:32:37.498410 3764 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [term 2 FOLLOWER]: Refusing update from remote peer e65dbad386c04dd381b04723f4a864e3: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260929 18:32:37.498962 4467 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260929 18:32:37.499450 4468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:37.499180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:33133 (local address 127.0.0.1:52732)
0929 18:32:37.499291 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:37.499360 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:33133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260929 18:32:37.499598 3932 consensus_peers.cc:487] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 -> Peer 1250cda9f69445ecadffa350f7d1790c (127.28.207.126:33133): Couldn't send request to peer 1250cda9f69445ecadffa350f7d1790c. Status: Network error: Client connection negotiation failed: client connection to 127.28.207.126:33133: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 18:32:37.500617 4466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e65dbad386c04dd381b04723f4a864e3. Latest consensus state: current_term: 2 leader_uuid: "e65dbad386c04dd381b04723f4a864e3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: false } } peers { permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 45571 } attrs { promote: false } } }
I20260929 18:32:37.500818 4466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:32:37.502175 4476 sys_catalog.cc:455] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "e65dbad386c04dd381b04723f4a864e3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: false } } peers { permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 45571 } attrs { promote: false } } }
I20260929 18:32:37.502339 4476 sys_catalog.cc:458] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3 [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:37.502555 4466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "e65dbad386c04dd381b04723f4a864e3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1250cda9f69445ecadffa350f7d1790c" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33133 } } peers { permanent_uuid: "9f296fd5b96e4975b59022170377b843" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 32877 } attrs { promote: false } } peers { permanent_uuid: "e65dbad386c04dd381b04723f4a864e3" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 45571 } attrs { promote: false } } }
I20260929 18:32:37.502701 4466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f296fd5b96e4975b59022170377b843 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:32:37.514221 4484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:37.509933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53292 (local address 127.28.207.65:37323)
0929 18:32:37.510229 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:37.510236 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:37.510330 (+ 94us) server_negotiation.cc:407] Connection header received
0929 18:32:37.510522 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:37.510525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:37.510589 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:37.510704 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:37.511248 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:37.511909 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:37.513177 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:37.513608 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:37.513733 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:37.513805 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:37.513999 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 18:32:37.514073 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":118,"threads_started":1}
I20260929 18:32:37.514221 4468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:37.509788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:37323 (local address 127.0.0.1:53292)
0929 18:32:37.510226 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:37.510247 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:37.510397 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:37.510750 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:37.510756 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:37.510781 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:37.510923 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:37.510930 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:37.512030 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:37.512036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:37.513059 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:37.513069 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:37.513730 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:37.513733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:37.513874 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:37.513893 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 18:32:37.513984 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 18:32:37.514074 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":41}
I20260929 18:32:37.515118 4056 tablet_service.cc:1467] Processing CreateTablet for tablet 073bfc6f20644bc1baec3bb4d35b6a75 (DEFAULT_TABLE table=test-table [id=753e334219814ebc9907710563e75e6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:37.515434 4056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 073bfc6f20644bc1baec3bb4d35b6a75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:37.516187 4469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:37.510836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:44677 (local address 127.0.0.1:57480)
0929 18:32:37.510967 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:37.510984 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:37.511122 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:37.511797 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:37.511802 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:37.511821 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:37.511971 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:37.511977 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:37.513232 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:37.513236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:37.514596 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:37.514607 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:37.515676 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:37.515680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:37.515803 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:37.515843 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 18:32:37.515919 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 18:32:37.516018 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":1}
I20260929 18:32:37.516491 4486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:37.511274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:36657 (local address 127.0.0.1:36512)
0929 18:32:37.511876 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:37.511891 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:37.511977 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:37.512504 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:37.512507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:37.512525 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:37.512619 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:37.512625 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:37.513787 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:37.513790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:37.515109 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:37.515118 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:37.516079 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:37.516082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:37.516187 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:37.516212 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 18:32:37.516260 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 18:32:37.516349 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":3,"thread_start_us":148,"threads_started":1}
I20260929 18:32:37.516865 4485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:37.511164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57480 (local address 127.28.207.66:44677)
0929 18:32:37.511491 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:37.511497 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:37.511514 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:32:37.511579 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:37.511582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:37.511640 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:37.512089 (+ 449us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:37.512208 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:37.513091 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:37.514762 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:37.515526 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:37.516462 (+ 936us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:37.516542 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:37.516588 (+ 46us) server_negotiation.cc:299] Negotiation successful
0929 18:32:37.516698 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":101,"threads_started":1}
I20260929 18:32:37.517000 4487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:37.511479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36512 (local address 127.28.207.67:36657)
0929 18:32:37.512224 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:37.512230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:37.512247 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:32:37.512304 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:37.512307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:37.512359 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:37.512724 (+ 365us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:37.512832 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:37.513646 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:37.515256 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:37.515927 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:37.516673 (+ 746us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:37.516736 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:37.516774 (+ 38us) server_negotiation.cc:299] Negotiation successful
0929 18:32:37.516866 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":71,"threads_started":1}
I20260929 18:32:37.517561 4331 tablet_service.cc:1467] Processing CreateTablet for tablet 073bfc6f20644bc1baec3bb4d35b6a75 (DEFAULT_TABLE table=test-table [id=753e334219814ebc9907710563e75e6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:37.517834 4331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 073bfc6f20644bc1baec3bb4d35b6a75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:37.517841 4191 tablet_service.cc:1467] Processing CreateTablet for tablet 073bfc6f20644bc1baec3bb4d35b6a75 (DEFAULT_TABLE table=test-table [id=753e334219814ebc9907710563e75e6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:37.518203 4191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 073bfc6f20644bc1baec3bb4d35b6a75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:37.518220 4488 tablet_bootstrap.cc:492] T 073bfc6f20644bc1baec3bb4d35b6a75 P 52f601bc57eb4420854cb9073fdd9caa: Bootstrap starting.
I20260929 18:32:37.519529 4488 tablet_bootstrap.cc:654] T 073bfc6f20644bc1baec3bb4d35b6a75 P 52f601bc57eb4420854cb9073fdd9caa: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:37.520135 4489 tablet_bootstrap.cc:492] T 073bfc6f20644bc1baec3bb4d35b6a75 P 37153dd37c404c03a93687dad1b4cdca: Bootstrap starting.
I20260929 18:32:37.521178 4489 tablet_bootstrap.cc:654] T 073bfc6f20644bc1baec3bb4d35b6a75 P 37153dd37c404c03a93687dad1b4cdca: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:37.522290 4492 tablet_bootstrap.cc:492] T 073bfc6f20644bc1baec3bb4d35b6a75 P af2018baa42e42b9b287a44255461376: Bootstrap starting.
I20260929 18:32:37.522688 4489 tablet_bootstrap.cc:492] T 073bfc6f20644bc1baec3bb4d35b6a75 P 37153dd37c404c03a93687dad1b4cdca: No bootstrap required, opened a new log
I20260929 18:32:37.522825 4489 ts_tablet_manager.cc:1397] T 073bfc6f20644bc1baec3bb4d35b6a75 P 37153dd37c404c03a93687dad1b4cdca: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:32:37.523267 4492 tablet_bootstrap.cc:654] T 073bfc6f20644bc1baec3bb4d35b6a75 P af2018baa42e42b9b287a44255461376: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:37.523434 4489 raft_consensus.cc:348] T 073bfc6f20644bc1baec3bb4d35b6a75 P 37153dd37c404c03a93687dad1b4cdca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f601bc57eb4420854cb9073fdd9caa" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 37323 } } peers { permanent_uuid: "af2018baa42e42b9b287a44255461376" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 44677 } } peers { permanent_uuid: "37153dd37c404c03a93687dad1b4cdca" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 36657 } }
I20260929 18:32:37.523586 4489 raft_consensus.cc:374] T 073bfc6f20644bc1baec3bb4d35b6a75 P 37153dd37c404c03a93687dad1b4cdca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:37.523613 4489 raft_consensus.cc:729] T 073bfc6f20644bc1baec3bb4d35b6a75 P 37153dd37c404c03a93687dad1b4cdca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37153dd37c404c03a93687dad1b4cdca, State: Initialized, Role: FOLLOWER
I20260929 18:32:37.523779 4489 consensus_queue.cc:260] T 073bfc6f20644bc1baec3bb4d35b6a75 P 37153dd37c404c03a93687dad1b4cdca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f601bc57eb4420854cb9073fdd9caa" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 37323 } } peers { permanent_uuid: "af2018baa42e42b9b287a44255461376" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 44677 } } peers { permanent_uuid: "37153dd37c404c03a93687dad1b4cdca" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 36657 } }
I20260929 18:32:37.524227 4489 ts_tablet_manager.cc:1428] T 073bfc6f20644bc1baec3bb4d35b6a75 P 37153dd37c404c03a93687dad1b4cdca: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:32:37.524753 4492 tablet_bootstrap.cc:492] T 073bfc6f20644bc1baec3bb4d35b6a75 P af2018baa42e42b9b287a44255461376: No bootstrap required, opened a new log
I20260929 18:32:37.524853 4492 ts_tablet_manager.cc:1397] T 073bfc6f20644bc1baec3bb4d35b6a75 P af2018baa42e42b9b287a44255461376: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:32:37.525277 4495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:37.524589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:33133 (local address 127.28.207.67:44027)
0929 18:32:37.525058 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:37.525145 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:33133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":113,"threads_started":1}
I20260929 18:32:37.525408 4492 raft_consensus.cc:348] T 073bfc6f20644bc1baec3bb4d35b6a75 P af2018baa42e42b9b287a44255461376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f601bc57eb4420854cb9073fdd9caa" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 37323 } } peers { permanent_uuid: "af2018baa42e42b9b287a44255461376" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 44677 } } peers { permanent_uuid: "37153dd37c404c03a93687dad1b4cdca" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 36657 } }
I20260929 18:32:37.525565 4492 raft_consensus.cc:374] T 073bfc6f20644bc1baec3bb4d35b6a75 P af2018baa42e42b9b287a44255461376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:37.525606 4492 raft_consensus.cc:729] T 073bfc6f20644bc1baec3bb4d35b6a75 P af2018baa42e42b9b287a44255461376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af2018baa42e42b9b287a44255461376, State: Initialized, Role: FOLLOWER
I20260929 18:32:37.525769 4492 consensus_queue.cc:260] T 073bfc6f20644bc1baec3bb4d35b6a75 P af2018baa42e42b9b287a44255461376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f601bc57eb4420854cb9073fdd9caa" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 37323 } } peers { permanent_uuid: "af2018baa42e42b9b287a44255461376" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 44677 } } peers { permanent_uuid: "37153dd37c404c03a93687dad1b4cdca" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 36657 } }
I20260929 18:32:37.526129 4492 ts_tablet_manager.cc:1428] T 073bfc6f20644bc1baec3bb4d35b6a75 P af2018baa42e42b9b287a44255461376: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 18:32:37.526240 4495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:37.525799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:33133 (local address 127.28.207.67:50495)
0929 18:32:37.526103 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:37.526160 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:33133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":108,"threads_started":1}
I20260929 18:32:37.527331 4498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:37.526847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:33133 (local address 127.28.207.66:48553)
0929 18:32:37.527154 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:37.527220 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:33133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":97,"threads_started":1}
I20260929 18:32:37.528312 4397 heartbeater.cc:503] Master 127.28.207.124:45571 was elected leader, sending a full tablet report...
I20260929 18:32:37.530689 4488 tablet_bootstrap.cc:492] T 073bfc6f20644bc1baec3bb4d35b6a75 P 52f601bc57eb4420854cb9073fdd9caa: No bootstrap required, opened a new log
I20260929 18:32:37.530809 4488 ts_tablet_manager.cc:1397] T 073bfc6f20644bc1baec3bb4d35b6a75 P 52f601bc57eb4420854cb9073fdd9caa: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20260929 18:32:37.531193 4259 heartbeater.cc:503] Master 127.28.207.124:45571 was elected leader, sending a full tablet report...
I20260929 18:32:37.531600 4488 raft_consensus.cc:348] T 073bfc6f20644bc1baec3bb4d35b6a75 P 52f601bc57eb4420854cb9073fdd9caa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f601bc57eb4420854cb9073fdd9caa" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 37323 } } peers { permanent_uuid: "af2018baa42e42b9b287a44255461376" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 44677 } } peers { permanent_uuid: "37153dd37c404c03a93687dad1b4cdca" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 36657 } }
I20260929 18:32:37.531769 4488 raft_consensus.cc:374] T 073bfc6f20644bc1baec3bb4d35b6a75 P 52f601bc57eb4420854cb9073fdd9caa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:37.531823 4488 raft_consensus.cc:729] T 073bfc6f20644bc1baec3bb4d35b6a75 P 52f601bc57eb4420854cb9073fdd9caa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52f601bc57eb4420854cb9073fdd9caa, State: Initialized, Role: FOLLOWER
I20260929 18:32:37.532022 4488 consensus_queue.cc:260] T 073bfc6f20644bc1baec3bb4d35b6a75 P 52f601bc57eb4420854cb9073fdd9caa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f601bc57eb4420854cb9073fdd9caa" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 37323 } } peers { permanent_uuid: "af2018baa42e42b9b287a44255461376" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 44677 } } peers { permanent_uuid: "37153dd37c404c03a93687dad1b4cdca" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 36657 } }
I20260929 18:32:37.532696 4498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:37.531528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:33133 (local address 127.28.207.66:44655)
0929 18:32:37.532535 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:37.532606 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:33133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260929 18:32:37.533564 4488 ts_tablet_manager.cc:1428] T 073bfc6f20644bc1baec3bb4d35b6a75 P 52f601bc57eb4420854cb9073fdd9caa: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 18:32:37.534034 4500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:37.532824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:33133 (local address 127.28.207.65:59299)
0929 18:32:37.533850 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:37.533916 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:33133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":956,"thread_start_us":126,"threads_started":1}
I20260929 18:32:37.534291 4122 heartbeater.cc:503] Master 127.28.207.124:45571 was elected leader, sending a full tablet report...
I20260929 18:32:37.534909 4501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:37.534473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:33133 (local address 127.28.207.65:43565)
0929 18:32:37.534702 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:37.534756 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:33133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":116,"threads_started":1}
I20260929 18:32:37.622138 4494 raft_consensus.cc:482] T 073bfc6f20644bc1baec3bb4d35b6a75 P 37153dd37c404c03a93687dad1b4cdca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:32:37.622293 4494 raft_consensus.cc:504] T 073bfc6f20644bc1baec3bb4d35b6a75 P 37153dd37c404c03a93687dad1b4cdca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f601bc57eb4420854cb9073fdd9caa" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 37323 } } peers { permanent_uuid: "af2018baa42e42b9b287a44255461376" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 44677 } } peers { permanent_uuid: "37153dd37c404c03a93687dad1b4cdca" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 36657 } }
I20260929 18:32:37.623634 4494 leader_election.cc:288] T 073bfc6f20644bc1baec3bb4d35b6a75 P 37153dd37c404c03a93687dad1b4cdca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52f601bc57eb4420854cb9073fdd9caa (127.28.207.65:37323), af2018baa42e42b9b287a44255461376 (127.28.207.66:44677)
I20260929 18:32:37.628067 4485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:37.624235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:35953 (local address 127.28.207.66:44677)
0929 18:32:37.624397 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:37.624401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:37.624420 (+ 19us) server_negotiation.cc:407] Connection header received
0929 18:32:37.624515 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:37.624518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:37.624572 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:37.624686 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:37.625110 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:37.625707 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:37.626972 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:37.627397 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:37.627506 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:37.627627 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:37.627795 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 18:32:37.627907 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 18:32:37.628078 4495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:37.624114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:44677 (local address 127.28.207.67:35953)
0929 18:32:37.624336 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:37.624350 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:37.624421 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:37.624751 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:37.624762 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:37.624786 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:37.624967 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:37.624975 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:37.625848 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:37.625851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:37.626777 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:37.626788 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:37.627511 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:37.627517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:37.627632 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:37.627660 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 18:32:37.627745 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 18:32:37.627893 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":1}
I20260929 18:32:37.628136 4484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:37.624030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:45273 (local address 127.28.207.65:37323)
0929 18:32:37.624208 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:37.624215 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:37.624247 (+ 32us) server_negotiation.cc:407] Connection header received
0929 18:32:37.624315 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:37.624318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:37.624376 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:37.624463 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:37.624928 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:37.625571 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:37.626918 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:37.627458 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:37.627598 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:37.627657 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:37.627928 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 18:32:37.628009 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 18:32:37.628120 4496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:37.623827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:37323 (local address 127.28.207.67:45273)
0929 18:32:37.624022 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:37.624039 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:37.624174 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:37.624517 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:37.624521 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:37.624554 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:37.624765 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:37.624775 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:37.625726 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:37.625732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:37.626745 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:37.626756 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:37.627633 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:37.627640 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:37.627783 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:37.627811 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 18:32:37.627897 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 18:32:37.627981 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74,"mutex_wait_us":27}
I20260929 18:32:37.628851 4075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073bfc6f20644bc1baec3bb4d35b6a75" candidate_uuid: "37153dd37c404c03a93687dad1b4cdca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f601bc57eb4420854cb9073fdd9caa" is_pre_election: true
I20260929 18:32:37.628950 4213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073bfc6f20644bc1baec3bb4d35b6a75" candidate_uuid: "37153dd37c404c03a93687dad1b4cdca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af2018baa42e42b9b287a44255461376" is_pre_election: true
I20260929 18:32:37.629081 4075 raft_consensus.cc:2454] T 073bfc6f20644bc1baec3bb4d35b6a75 P 52f601bc57eb4420854cb9073fdd9caa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37153dd37c404c03a93687dad1b4cdca in term 0.
I20260929 18:32:37.629295 4213 raft_consensus.cc:2454] T 073bfc6f20644bc1baec3bb4d35b6a75 P af2018baa42e42b9b287a44255461376 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37153dd37c404c03a93687dad1b4cdca in term 0.
I20260929 18:32:37.629539 4285 leader_election.cc:302] T 073bfc6f20644bc1baec3bb4d35b6a75 P 37153dd37c404c03a93687dad1b4cdca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37153dd37c404c03a93687dad1b4cdca, 52f601bc57eb4420854cb9073fdd9caa; no voters:
I20260929 18:32:37.629978 4494 raft_consensus.cc:2781] T 073bfc6f20644bc1baec3bb4d35b6a75 P 37153dd37c404c03a93687dad1b4cdca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:32:37.630072 4494 raft_consensus.cc:482] T 073bfc6f20644bc1baec3bb4d35b6a75 P 37153dd37c404c03a93687dad1b4cdca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:32:37.630106 4494 raft_consensus.cc:3037] T 073bfc6f20644bc1baec3bb4d35b6a75 P 37153dd37c404c03a93687dad1b4cdca [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:37.631531 4494 raft_consensus.cc:504] T 073bfc6f20644bc1baec3bb4d35b6a75 P 37153dd37c404c03a93687dad1b4cdca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f601bc57eb4420854cb9073fdd9caa" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 37323 } } peers { permanent_uuid: "af2018baa42e42b9b287a44255461376" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 44677 } } peers { permanent_uuid: "37153dd37c404c03a93687dad1b4cdca" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 36657 } }
I20260929 18:32:37.632035 4494 leader_election.cc:288] T 073bfc6f20644bc1baec3bb4d35b6a75 P 37153dd37c404c03a93687dad1b4cdca [CANDIDATE]: Term 1 election: Requested vote from peers 52f601bc57eb4420854cb9073fdd9caa (127.28.207.65:37323), af2018baa42e42b9b287a44255461376 (127.28.207.66:44677)
I20260929 18:32:37.632539 4075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073bfc6f20644bc1baec3bb4d35b6a75" candidate_uuid: "37153dd37c404c03a93687dad1b4cdca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52f601bc57eb4420854cb9073fdd9caa"
I20260929 18:32:37.632540 4213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "073bfc6f20644bc1baec3bb4d35b6a75" candidate_uuid: "37153dd37c404c03a93687dad1b4cdca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af2018baa42e42b9b287a44255461376"
I20260929 18:32:37.632705 4075 raft_consensus.cc:3037] T 073bfc6f20644bc1baec3bb4d35b6a75 P 52f601bc57eb4420854cb9073fdd9caa [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:37.632730 4213 raft_consensus.cc:3037] T 073bfc6f20644bc1baec3bb4d35b6a75 P af2018baa42e42b9b287a44255461376 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:37.633844 4075 raft_consensus.cc:2454] T 073bfc6f20644bc1baec3bb4d35b6a75 P 52f601bc57eb4420854cb9073fdd9caa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37153dd37c404c03a93687dad1b4cdca in term 1.
I20260929 18:32:37.633944 4213 raft_consensus.cc:2454] T 073bfc6f20644bc1baec3bb4d35b6a75 P af2018baa42e42b9b287a44255461376 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37153dd37c404c03a93687dad1b4cdca in term 1.
I20260929 18:32:37.634239 4285 leader_election.cc:302] T 073bfc6f20644bc1baec3bb4d35b6a75 P 37153dd37c404c03a93687dad1b4cdca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37153dd37c404c03a93687dad1b4cdca, 52f601bc57eb4420854cb9073fdd9caa; no voters:
I20260929 18:32:37.634455 4494 raft_consensus.cc:2781] T 073bfc6f20644bc1baec3bb4d35b6a75 P 37153dd37c404c03a93687dad1b4cdca [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:32:37.634739 4494 raft_consensus.cc:686] T 073bfc6f20644bc1baec3bb4d35b6a75 P 37153dd37c404c03a93687dad1b4cdca [term 1 LEADER]: Becoming Leader. State: Replica: 37153dd37c404c03a93687dad1b4cdca, State: Running, Role: LEADER
I20260929 18:32:37.635068 4494 consensus_queue.cc:237] T 073bfc6f20644bc1baec3bb4d35b6a75 P 37153dd37c404c03a93687dad1b4cdca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f601bc57eb4420854cb9073fdd9caa" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 37323 } } peers { permanent_uuid: "af2018baa42e42b9b287a44255461376" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 44677 } } peers { permanent_uuid: "37153dd37c404c03a93687dad1b4cdca" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 36657 } }
I20260929 18:32:37.635728 4496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:37.635401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:33133 (local address 127.28.207.67:60031)
0929 18:32:37.635576 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:37.635645 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:33133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 18:32:37.638588 3951 catalog_manager.cc:5495] T 073bfc6f20644bc1baec3bb4d35b6a75 P 37153dd37c404c03a93687dad1b4cdca reported cstate change: term changed from 0 to 1, leader changed from <none> to 37153dd37c404c03a93687dad1b4cdca (127.28.207.67). New cstate: current_term: 1 leader_uuid: "37153dd37c404c03a93687dad1b4cdca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52f601bc57eb4420854cb9073fdd9caa" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 37323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af2018baa42e42b9b287a44255461376" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 44677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37153dd37c404c03a93687dad1b4cdca" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 36657 } health_report { overall_health: HEALTHY } } }
I20260929 18:32:37.677718 4487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:37.673827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36516 (local address 127.28.207.67:36657)
0929 18:32:37.673984 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:37.673990 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:37.674007 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:32:37.674185 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:37.674188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:37.674260 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:37.674430 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:32:37.674684 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:37.675224 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:37.676192 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:37.676409 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:37.676609 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:37.676863 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:32:37.677318 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:32:37.677441 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:37.677509 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 18:32:37.677569 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 18:32:37.681162 4075 raft_consensus.cc:1261] T 073bfc6f20644bc1baec3bb4d35b6a75 P 52f601bc57eb4420854cb9073fdd9caa [term 1 FOLLOWER]: Refusing update from remote peer 37153dd37c404c03a93687dad1b4cdca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:32:37.681694 4494 consensus_queue.cc:1035] T 073bfc6f20644bc1baec3bb4d35b6a75 P 37153dd37c404c03a93687dad1b4cdca [LEADER]: Connected to new peer: Peer: permanent_uuid: "52f601bc57eb4420854cb9073fdd9caa" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 37323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:32:37.681802 4213 raft_consensus.cc:1261] T 073bfc6f20644bc1baec3bb4d35b6a75 P af2018baa42e42b9b287a44255461376 [term 1 FOLLOWER]: Refusing update from remote peer 37153dd37c404c03a93687dad1b4cdca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:32:37.682538 4494 consensus_queue.cc:1035] T 073bfc6f20644bc1baec3bb4d35b6a75 P 37153dd37c404c03a93687dad1b4cdca [LEADER]: Connected to new peer: Peer: permanent_uuid: "af2018baa42e42b9b287a44255461376" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 44677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:32:37.683650 4501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:37.683140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:33133 (local address 127.28.207.65:56839)
0929 18:32:37.683511 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:37.683572 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:33133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301}
I20260929 18:32:37.685722 4498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:37.685483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:33133 (local address 127.28.207.66:60093)
0929 18:32:37.685578 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:37.685638 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:33133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260929 18:32:37.687592 4508 mvcc.cc:205] Tried to move back new op lower bound from 7334734879456083968 to 7334734879276244992. Current Snapshot: MvccSnapshot[applied={T|T < 7334734879456083968}]
I20260929 18:32:37.688405 4496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:37.688167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:33133 (local address 127.28.207.67:54627)
0929 18:32:37.688296 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:37.688348 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:33133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 18:32:37.692919 4496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:37.692661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:33133 (local address 127.28.207.67:46809)
0929 18:32:37.692782 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:37.692846 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:33133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 18:32:37.692941 4498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:37.691200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:33133 (local address 127.28.207.66:45721)
0929 18:32:37.692715 (+ 1515us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:37.692808 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:33133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1415}
I20260929 18:32:37.697763 4501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:37.697350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:33133 (local address 127.28.207.65:60383)
0929 18:32:37.697560 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:37.697637 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:33133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260929 18:32:37.698222 3951 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:32:37.698443 3951 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:32:37.703331 3951 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e65dbad386c04dd381b04723f4a864e3: Sending DeleteTablet for 3 replicas of tablet 073bfc6f20644bc1baec3bb4d35b6a75
I20260929 18:32:37.704070 4056 tablet_service.cc:1514] Processing DeleteTablet for tablet 073bfc6f20644bc1baec3bb4d35b6a75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:37 UTC) from {username='oryx', principal='oryx/127.28.207.124@KRBTEST.COM'} at 127.0.0.1:53292
I20260929 18:32:37.704393 4191 tablet_service.cc:1514] Processing DeleteTablet for tablet 073bfc6f20644bc1baec3bb4d35b6a75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:37 UTC) from {username='oryx', principal='oryx/127.28.207.124@KRBTEST.COM'} at 127.0.0.1:57480
I20260929 18:32:37.704531 4331 tablet_service.cc:1514] Processing DeleteTablet for tablet 073bfc6f20644bc1baec3bb4d35b6a75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:37 UTC) from {username='oryx', principal='oryx/127.28.207.124@KRBTEST.COM'} at 127.0.0.1:36512
I20260929 18:32:37.704901 4516 tablet_replica.cc:330] T 073bfc6f20644bc1baec3bb4d35b6a75 P af2018baa42e42b9b287a44255461376: stopping tablet replica
I20260929 18:32:37.705060 4516 raft_consensus.cc:2229] T 073bfc6f20644bc1baec3bb4d35b6a75 P af2018baa42e42b9b287a44255461376 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:32:37.705227 4516 raft_consensus.cc:2258] T 073bfc6f20644bc1baec3bb4d35b6a75 P af2018baa42e42b9b287a44255461376 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:37.706103 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 3898
I20260929 18:32:37.707345 4516 ts_tablet_manager.cc:1905] T 073bfc6f20644bc1baec3bb4d35b6a75 P af2018baa42e42b9b287a44255461376: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:32:37.711298 4516 ts_tablet_manager.cc:1918] T 073bfc6f20644bc1baec3bb4d35b6a75 P af2018baa42e42b9b287a44255461376: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:32:37.711429 4516 log.cc:1196] T 073bfc6f20644bc1baec3bb4d35b6a75 P af2018baa42e42b9b287a44255461376: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-1/wal/wals/073bfc6f20644bc1baec3bb4d35b6a75
I20260929 18:32:37.711730 4517 tablet_replica.cc:330] T 073bfc6f20644bc1baec3bb4d35b6a75 P 37153dd37c404c03a93687dad1b4cdca: stopping tablet replica
I20260929 18:32:37.711908 4517 raft_consensus.cc:2229] T 073bfc6f20644bc1baec3bb4d35b6a75 P 37153dd37c404c03a93687dad1b4cdca [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:32:37.712057 4516 ts_tablet_manager.cc:1939] T 073bfc6f20644bc1baec3bb4d35b6a75 P af2018baa42e42b9b287a44255461376: Deleting consensus metadata
I20260929 18:32:37.712332 4517 raft_consensus.cc:2258] T 073bfc6f20644bc1baec3bb4d35b6a75 P 37153dd37c404c03a93687dad1b4cdca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:37.713129 3933 catalog_manager.cc:4841] TS af2018baa42e42b9b287a44255461376 (127.28.207.66:44677): tablet 073bfc6f20644bc1baec3bb4d35b6a75 (table test-table [id=753e334219814ebc9907710563e75e6f]) successfully deleted
I20260929 18:32:37.713326 4517 ts_tablet_manager.cc:1905] T 073bfc6f20644bc1baec3bb4d35b6a75 P 37153dd37c404c03a93687dad1b4cdca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:32:37.716926 4517 ts_tablet_manager.cc:1918] T 073bfc6f20644bc1baec3bb4d35b6a75 P 37153dd37c404c03a93687dad1b4cdca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:32:37.717023 4517 log.cc:1196] T 073bfc6f20644bc1baec3bb4d35b6a75 P 37153dd37c404c03a93687dad1b4cdca: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790706718260899-29501-0/minicluster-data/ts-2/wal/wals/073bfc6f20644bc1baec3bb4d35b6a75
I20260929 18:32:37.717346 4517 ts_tablet_manager.cc:1939] T 073bfc6f20644bc1baec3bb4d35b6a75 P 37153dd37c404c03a93687dad1b4cdca: Deleting consensus metadata
I20260929 18:32:37.718181 3932 catalog_manager.cc:4841] TS 37153dd37c404c03a93687dad1b4cdca (127.28.207.67:36657): tablet 073bfc6f20644bc1baec3bb4d35b6a75 (table test-table [id=753e334219814ebc9907710563e75e6f]) successfully deleted
W20260929 18:32:37.718881 4486 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:37.718394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:37323 (local address 127.0.0.1:53306)
0929 18:32:37.718515 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:37.718533 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:37.718626 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:37.718735 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.65:37323: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 18:32:37.719136 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 4130
W20260929 18:32:37.722108 3931 catalog_manager.cc:4584] TS 52f601bc57eb4420854cb9073fdd9caa (127.28.207.65:37323): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 073bfc6f20644bc1baec3bb4d35b6a75: Network error: Client connection negotiation failed: client connection to 127.28.207.65:37323: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260929 18:32:37.732255 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 4266
I20260929 18:32:37.739478 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 3615
I20260929 18:32:37.746675 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 3718
2026-09-29T18:32:37Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5857 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:32:37 dist-test-slave-qcpv krb5kdc[4520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:32:37 dist-test-slave-qcpv krb5kdc[4520](info): set up 2 sockets
Sep 29 18:32:37 dist-test-slave-qcpv krb5kdc[4520](info): commencing operation
krb5kdc: starting...
W20260929 18:32:38.844719 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:32:38 dist-test-slave-qcpv krb5kdc[4520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:32:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:32:38Z Disabled control of system clock
WARNING: no policy specified for oryx/127.28.207.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.207.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.207.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.126.keytab.
Entry for principal oryx/127.28.207.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.126.keytab.
I20260929 18:32:39.000252 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:37575
--webserver_interface=127.28.207.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.126.keytab
--principal=oryx/127.28.207.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:45783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:37575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:39.126955 4533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:39.127286 4533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:39.127434 4533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:39.132757 4533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:32:39.132870 4533 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 18:32:39.132905 4533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:39.132938 4533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:39.132956 4533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:32:39.132973 4533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:32:39.139057 4533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:45783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:37575
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.126.keytab
--principal=oryx/127.28.207.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:37575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:39.140167 4533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:39.141194 4533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:39.151713 4538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:39.151968 4539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:39.151703 4541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:39.152374 4533 server_base.cc:1034] running on GCE node
I20260929 18:32:39.153020 4533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:39.154220 4533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:39.155432 4533 hybrid_clock.cc:648] HybridClock initialized: now 1790706759155364 us; error 86 us; skew 500 ppm
Sep 29 18:32:39 dist-test-slave-qcpv krb5kdc[4520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706759, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.207.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:39.167433 4533 init.cc:377] Logged in from keytab as oryx/127.28.207.126@KRBTEST.COM (short username oryx)
I20260929 18:32:39.168622 4533 webserver.cc:460] Webserver started at http://127.28.207.126:43307/ using document root <none> and password file <none>
I20260929 18:32:39.169246 4533 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:39.169348 4533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:39.169590 4533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:39.171846 4533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "bf94f8dce239472093ae97936b750185"
format_stamp: "Formatted at 2026-09-29 18:32:39 on dist-test-slave-qcpv"
server_key: "e19af7eee21e565b9b0ae031eade87ef"
server_key_iv: "6d9e878d090e769472e678c25d352a7a"
server_key_version: "encryptionkey@0"
I20260929 18:32:39.172423 4533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "bf94f8dce239472093ae97936b750185"
format_stamp: "Formatted at 2026-09-29 18:32:39 on dist-test-slave-qcpv"
server_key: "e19af7eee21e565b9b0ae031eade87ef"
server_key_iv: "6d9e878d090e769472e678c25d352a7a"
server_key_version: "encryptionkey@0"
I20260929 18:32:39.176720 4533 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:32:39.180181 4548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:39.181874 4533 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 18:32:39.182070 4533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "bf94f8dce239472093ae97936b750185"
format_stamp: "Formatted at 2026-09-29 18:32:39 on dist-test-slave-qcpv"
server_key: "e19af7eee21e565b9b0ae031eade87ef"
server_key_iv: "6d9e878d090e769472e678c25d352a7a"
server_key_version: "encryptionkey@0"
I20260929 18:32:39.182206 4533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:39.230682 4533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:39.234439 4533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:39.234665 4533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:39.243990 4533 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:37575
I20260929 18:32:39.244015 4610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:37575 every 8 connection(s)
I20260929 18:32:39.245085 4533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:32:39.249255 4611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:39.251191 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 4533
I20260929 18:32:39.251328 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:32:39.251643 29501 external_mini_cluster.cc:1428] Setting key cbb0ddc4c8347c71b120ca1bc0f4adc5
I20260929 18:32:39.258364 4611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf94f8dce239472093ae97936b750185: Bootstrap starting.
I20260929 18:32:39.262036 4611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf94f8dce239472093ae97936b750185: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:39.263211 4611 log.cc:824] T 00000000000000000000000000000000 P bf94f8dce239472093ae97936b750185: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:32:39 dist-test-slave-qcpv krb5kdc[4520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM
I20260929 18:32:39.266181 4611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf94f8dce239472093ae97936b750185: No bootstrap required, opened a new log
I20260929 18:32:39.269582 4611 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf94f8dce239472093ae97936b750185 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf94f8dce239472093ae97936b750185" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 37575 } }
I20260929 18:32:39.269899 4611 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf94f8dce239472093ae97936b750185 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:39.269963 4611 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf94f8dce239472093ae97936b750185 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf94f8dce239472093ae97936b750185, State: Initialized, Role: FOLLOWER
I20260929 18:32:39.270479 4611 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf94f8dce239472093ae97936b750185 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf94f8dce239472093ae97936b750185" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 37575 } }
I20260929 18:32:39.270649 4611 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf94f8dce239472093ae97936b750185 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:32:39.270741 4611 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf94f8dce239472093ae97936b750185 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:32:39.270910 4611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf94f8dce239472093ae97936b750185 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:39.271584 4614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:39.253424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32950 (local address 127.28.207.126:37575)
0929 18:32:39.254071 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:39.254095 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:39.254146 (+ 51us) server_negotiation.cc:407] Connection header received
0929 18:32:39.255459 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:39.255494 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:39.256115 (+ 621us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:39.256598 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:39.257417 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:39.258992 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:39.260021 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:39.260771 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:39.260982 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:39.264468 (+ 3486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:39.264475 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:39.264494 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:39.264531 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:39.267376 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:39.267982 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:39.267989 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:39.267995 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:39.268136 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:39.268483 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:39.268488 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:39.268490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:39.269048 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:39.269327 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:39.269797 (+ 470us) server_negotiation.cc:299] Negotiation successful
0929 18:32:39.270310 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":135,"threads_started":1}
I20260929 18:32:39.272181 4611 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf94f8dce239472093ae97936b750185 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf94f8dce239472093ae97936b750185" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 37575 } }
I20260929 18:32:39.272568 4611 leader_election.cc:302] T 00000000000000000000000000000000 P bf94f8dce239472093ae97936b750185 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf94f8dce239472093ae97936b750185; no voters:
I20260929 18:32:39.273064 4611 leader_election.cc:288] T 00000000000000000000000000000000 P bf94f8dce239472093ae97936b750185 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:32:39.273188 4616 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf94f8dce239472093ae97936b750185 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:32:39.273433 4616 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf94f8dce239472093ae97936b750185 [term 1 LEADER]: Becoming Leader. State: Replica: bf94f8dce239472093ae97936b750185, State: Running, Role: LEADER
I20260929 18:32:39.273759 4616 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf94f8dce239472093ae97936b750185 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf94f8dce239472093ae97936b750185" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 37575 } }
I20260929 18:32:39.274400 4611 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf94f8dce239472093ae97936b750185 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:32:39.275254 4618 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf94f8dce239472093ae97936b750185 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf94f8dce239472093ae97936b750185. Latest consensus state: current_term: 1 leader_uuid: "bf94f8dce239472093ae97936b750185" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf94f8dce239472093ae97936b750185" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 37575 } } }
I20260929 18:32:39.275393 4618 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf94f8dce239472093ae97936b750185 [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:39.275851 4617 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf94f8dce239472093ae97936b750185 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf94f8dce239472093ae97936b750185" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf94f8dce239472093ae97936b750185" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 37575 } } }
I20260929 18:32:39.275952 4617 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf94f8dce239472093ae97936b750185 [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:39.276284 4624 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:32:39.279201 4624 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:32:39.285660 4624 catalog_manager.cc:1347] Generated new cluster ID: 01c00671569c4b2d84a5dfb50b268bf4
I20260929 18:32:39.285780 4624 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:32:39.296113 4624 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:32:39.297215 4624 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:32:39.307752 4624 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf94f8dce239472093ae97936b750185: Generated new TSK 0
I20260929 18:32:39.308576 4624 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.28.207.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.207.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.207.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.65.keytab.
Entry for principal oryx/127.28.207.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.65.keytab.
I20260929 18:32:39.342443 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.65:0
--local_ip_for_outbound_sockets=127.28.207.65
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:37575
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.65.keytab
--principal=oryx/127.28.207.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:45783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:39.464370 4637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:39.464684 4637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:39.464733 4637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:39.468876 4637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:39.468950 4637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:39.468990 4637 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:32:39.469024 4637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.65
I20260929 18:32:39.474551 4637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:45783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.65.keytab
--principal=oryx/127.28.207.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.207.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.207.126:37575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.65
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:39.475814 4637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:39.476855 4637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:39.487859 4642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:39.487859 4645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:39.488116 4643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:39.488611 4637 server_base.cc:1034] running on GCE node
I20260929 18:32:39.489085 4637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:39.489823 4637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:39.491014 4637 hybrid_clock.cc:648] HybridClock initialized: now 1790706759490965 us; error 69 us; skew 500 ppm
Sep 29 18:32:39 dist-test-slave-qcpv krb5kdc[4520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706759, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.207.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:39.494465 4637 init.cc:377] Logged in from keytab as oryx/127.28.207.65@KRBTEST.COM (short username oryx)
I20260929 18:32:39.495533 4637 webserver.cc:460] Webserver started at http://127.28.207.65:44643/ using document root <none> and password file <none>
I20260929 18:32:39.496169 4637 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:39.496261 4637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:39.496490 4637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:39.498761 4637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-0/data/instance:
uuid: "4eb628a7dcc2441aa375002c139dcace"
format_stamp: "Formatted at 2026-09-29 18:32:39 on dist-test-slave-qcpv"
server_key: "85542426b6277e4b0663c6f774805e0d"
server_key_iv: "893ea51dd875a36de569a759b25bb7b3"
server_key_version: "encryptionkey@0"
I20260929 18:32:39.499426 4637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance:
uuid: "4eb628a7dcc2441aa375002c139dcace"
format_stamp: "Formatted at 2026-09-29 18:32:39 on dist-test-slave-qcpv"
server_key: "85542426b6277e4b0663c6f774805e0d"
server_key_iv: "893ea51dd875a36de569a759b25bb7b3"
server_key_version: "encryptionkey@0"
I20260929 18:32:39.503595 4637 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:32:39.506464 4652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:39.508090 4637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 18:32:39.508288 4637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-0/wal
uuid: "4eb628a7dcc2441aa375002c139dcace"
format_stamp: "Formatted at 2026-09-29 18:32:39 on dist-test-slave-qcpv"
server_key: "85542426b6277e4b0663c6f774805e0d"
server_key_iv: "893ea51dd875a36de569a759b25bb7b3"
server_key_version: "encryptionkey@0"
I20260929 18:32:39.508409 4637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:39.529279 4637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:39.532754 4637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:39.532974 4637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:39.534631 4637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:39.534713 4637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:39.534767 4637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:39.534789 4637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:39.546413 4637 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.65:38705
I20260929 18:32:39.546438 4767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.65:38705 every 8 connection(s)
Sep 29 18:32:39 dist-test-slave-qcpv krb5kdc[4520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706759, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.207.65@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM
I20260929 18:32:39.547592 4637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
I20260929 18:32:39.550707 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 4637
I20260929 18:32:39.550948 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance
I20260929 18:32:39.551327 29501 external_mini_cluster.cc:1428] Setting key af7e0e0c9c0d54612c49ecdd5eaa7427
I20260929 18:32:39.552191 4614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:39.535583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:44631 (local address 127.28.207.126:37575)
0929 18:32:39.535740 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:39.535744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:39.536915 (+ 1171us) server_negotiation.cc:407] Connection header received
0929 18:32:39.538528 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:39.538532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:39.538625 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:39.538773 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:39.540718 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:39.541424 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:39.543183 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:39.543504 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:39.543638 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:39.547582 (+ 3944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:39.547586 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:39.547588 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:39.547627 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:39.549867 (+ 2240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:39.550452 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:39.550457 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:39.550459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:39.550545 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:39.550942 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:39.550945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:39.550947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:39.551201 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:39.551306 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:39.551862 (+ 556us) server_negotiation.cc:299] Negotiation successful
0929 18:32:39.552009 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:32:39.553278 4661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:39.536057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:37575 (local address 127.28.207.65:44631)
0929 18:32:39.536697 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:39.536761 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:39.538158 (+ 1397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:39.539120 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:39.539129 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:39.539653 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:39.540499 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:39.540518 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:39.541600 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:39.541605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:39.542968 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:39.542980 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:39.543700 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:39.543704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:39.543933 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:39.544927 (+ 994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:39.544955 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:39.547299 (+ 2344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:39.550016 (+ 2717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:39.550023 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:39.550038 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:39.550333 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:39.550634 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:39.550639 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:39.550641 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:39.550786 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:39.551302 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:39.551309 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:39.551622 (+ 313us) client_negotiation.cc:769] Sending connection context
0929 18:32:39.551804 (+ 182us) client_negotiation.cc:240] Negotiation successful
0929 18:32:39.552144 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":152,"threads_started":1}
I20260929 18:32:39.554949 4768 heartbeater.cc:347] Connected to a master server at 127.28.207.126:37575
I20260929 18:32:39.555197 4768 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:39.556088 4768 heartbeater.cc:511] Master 127.28.207.126:37575 requested a full tablet report, sending...
I20260929 18:32:39.558215 4565 ts_manager.cc:194] Registered new tserver with Master: 4eb628a7dcc2441aa375002c139dcace (127.28.207.65:38705)
I20260929 18:32:39.559896 4565 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.207.65@KRBTEST.COM'} at 127.28.207.65:44631
WARNING: no policy specified for oryx/127.28.207.66@KRBTEST.COM; defaulting to no policy
I20260929 18:32:39.568400 4775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:39.557356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:37575 (local address 127.28.207.65:37827)
0929 18:32:39.557726 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:39.557746 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:39.558994 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:39.559317 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:39.559320 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:39.559554 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:39.559693 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:39.559699 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:39.560592 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:39.560595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:39.561538 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:39.561551 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:39.562383 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:39.562387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:39.562503 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:39.563311 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:39.563326 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:39.563767 (+ 441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:39.566278 (+ 2511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:39.566281 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:39.566283 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:39.566553 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:39.566987 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:39.566990 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:39.566991 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:39.567078 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:39.567781 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:39.567792 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:39.567893 (+ 101us) client_negotiation.cc:769] Sending connection context
0929 18:32:39.567999 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 18:32:39.568221 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":1,"thread_start_us":135,"threads_started":1}
I20260929 18:32:39.568598 4614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:39.557503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:37827 (local address 127.28.207.126:37575)
0929 18:32:39.557658 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:39.557663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:39.557861 (+ 198us) server_negotiation.cc:407] Connection header received
0929 18:32:39.559125 (+ 1264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:39.559128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:39.559213 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:39.559299 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:39.559807 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:39.560471 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:39.561715 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:39.562214 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:39.562378 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:39.563956 (+ 1578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:39.563962 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:39.563970 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:39.564011 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:39.566079 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:39.566691 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:39.566695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:39.566698 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:39.566809 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:39.567262 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:39.567268 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:39.567274 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:39.567632 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:39.567768 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:39.568118 (+ 350us) server_negotiation.cc:299] Negotiation successful
0929 18:32:39.568299 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.207.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.207.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.66.keytab.
Entry for principal oryx/127.28.207.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.66.keytab.
I20260929 18:32:39.591862 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.66:0
--local_ip_for_outbound_sockets=127.28.207.66
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:37575
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.66.keytab
--principal=oryx/127.28.207.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:45783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:39.710398 4777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:39.710738 4777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:39.710824 4777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:39.715195 4777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:39.715301 4777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:39.715377 4777 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:32:39.715459 4777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.66
I20260929 18:32:39.721213 4777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:45783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.66.keytab
--principal=oryx/127.28.207.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.207.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.207.126:37575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.66
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:39.722379 4777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:39.723443 4777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:39.734402 4785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:39.734558 4783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:39.734558 4782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:39.734745 4777 server_base.cc:1034] running on GCE node
I20260929 18:32:39.735467 4777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:39.736308 4777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:39.737552 4777 hybrid_clock.cc:648] HybridClock initialized: now 1790706759737494 us; error 80 us; skew 500 ppm
Sep 29 18:32:39 dist-test-slave-qcpv krb5kdc[4520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706759, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.207.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:39.741480 4777 init.cc:377] Logged in from keytab as oryx/127.28.207.66@KRBTEST.COM (short username oryx)
I20260929 18:32:39.742735 4777 webserver.cc:460] Webserver started at http://127.28.207.66:41453/ using document root <none> and password file <none>
I20260929 18:32:39.743528 4777 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:39.743637 4777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:39.743916 4777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:39.747063 4777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-1/data/instance:
uuid: "4ee346edbe3a4443bb23cccccd8b9f75"
format_stamp: "Formatted at 2026-09-29 18:32:39 on dist-test-slave-qcpv"
server_key: "061f53491b8c191c14098d7a182611a2"
server_key_iv: "92d754fd37bb6a0f7a15684870c26501"
server_key_version: "encryptionkey@0"
I20260929 18:32:39.747879 4777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance:
uuid: "4ee346edbe3a4443bb23cccccd8b9f75"
format_stamp: "Formatted at 2026-09-29 18:32:39 on dist-test-slave-qcpv"
server_key: "061f53491b8c191c14098d7a182611a2"
server_key_iv: "92d754fd37bb6a0f7a15684870c26501"
server_key_version: "encryptionkey@0"
I20260929 18:32:39.753417 4777 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 18:32:39.757563 4792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:39.759550 4777 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:32:39.759744 4777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-1/wal
uuid: "4ee346edbe3a4443bb23cccccd8b9f75"
format_stamp: "Formatted at 2026-09-29 18:32:39 on dist-test-slave-qcpv"
server_key: "061f53491b8c191c14098d7a182611a2"
server_key_iv: "92d754fd37bb6a0f7a15684870c26501"
server_key_version: "encryptionkey@0"
I20260929 18:32:39.759908 4777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:39.772859 4777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:39.777804 4777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:39.778043 4777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:39.779688 4777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:39.779757 4777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:39.779821 4777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:39.779935 4777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:32:39 dist-test-slave-qcpv krb5kdc[4520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706759, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.207.66@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM
I20260929 18:32:39.793357 4777 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.66:38545
I20260929 18:32:39.793460 4907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.66:38545 every 8 connection(s)
I20260929 18:32:39.794433 4777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
I20260929 18:32:39.796988 4614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:39.780747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:39927 (local address 127.28.207.126:37575)
0929 18:32:39.780951 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:39.780958 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:39.781835 (+ 877us) server_negotiation.cc:407] Connection header received
0929 18:32:39.783041 (+ 1206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:39.783046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:39.783109 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:39.783198 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:39.784904 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:39.785707 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:39.788103 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:39.788489 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:39.788619 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:39.792485 (+ 3866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:39.792488 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:39.792491 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:39.792520 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:39.794268 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:39.794951 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:39.794957 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:39.794958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:39.795094 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:39.795520 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:39.795524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:39.795525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:39.795786 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:39.795874 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:39.796616 (+ 742us) server_negotiation.cc:299] Negotiation successful
0929 18:32:39.796805 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 18:32:39.797094 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 4777
I20260929 18:32:39.797220 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance
I20260929 18:32:39.797551 29501 external_mini_cluster.cc:1428] Setting key 2c35796331a633363e23a750320c3b88
I20260929 18:32:39.797878 4801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:39.781163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:37575 (local address 127.28.207.66:39927)
0929 18:32:39.781646 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:39.781701 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:39.782617 (+ 916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:39.783409 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:39.783419 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:39.783946 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:39.784746 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:39.784772 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:39.785942 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:39.785951 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:39.787899 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:39.787924 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:39.788704 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:39.788711 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:39.788922 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:39.789802 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:39.789820 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:39.792267 (+ 2447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:39.794408 (+ 2141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:39.794415 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:39.794431 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:39.794778 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:39.795213 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:39.795216 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:39.795217 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:39.795391 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:39.795910 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:39.795916 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:39.796182 (+ 266us) client_negotiation.cc:769] Sending connection context
0929 18:32:39.796365 (+ 183us) client_negotiation.cc:240] Negotiation successful
0929 18:32:39.796679 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":137,"threads_started":1}
I20260929 18:32:39.799403 4908 heartbeater.cc:347] Connected to a master server at 127.28.207.126:37575
I20260929 18:32:39.799695 4908 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:39.800485 4908 heartbeater.cc:511] Master 127.28.207.126:37575 requested a full tablet report, sending...
I20260929 18:32:39.802048 4565 ts_manager.cc:194] Registered new tserver with Master: 4ee346edbe3a4443bb23cccccd8b9f75 (127.28.207.66:38545)
I20260929 18:32:39.803102 4565 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.207.66@KRBTEST.COM'} at 127.28.207.66:39927
WARNING: no policy specified for oryx/127.28.207.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.207.67@KRBTEST.COM" created.
I20260929 18:32:39.813998 4915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:39.802320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:37575 (local address 127.28.207.66:34249)
0929 18:32:39.802699 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:39.802721 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:39.802920 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:39.803236 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:39.803240 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:39.803560 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:39.803701 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:39.803712 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:39.805868 (+ 2156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:39.805872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:39.806550 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:39.806558 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:39.807292 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:39.807296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:39.807419 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:39.808224 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:39.808247 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:39.808807 (+ 560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:39.811875 (+ 3068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:39.811880 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:39.811882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:39.812238 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:39.812555 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:39.812557 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:39.812559 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:39.812644 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:39.813301 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:39.813305 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:39.813431 (+ 126us) client_negotiation.cc:769] Sending connection context
0929 18:32:39.813551 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 18:32:39.813767 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":125,"threads_started":1}
I20260929 18:32:39.813988 4614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:39.802739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:34249 (local address 127.28.207.126:37575)
0929 18:32:39.802921 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:39.802927 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:39.802947 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:32:39.803018 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:39.803021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:39.803079 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:39.803202 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:39.803852 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:39.804700 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:39.806726 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:39.807144 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:39.807284 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:39.809046 (+ 1762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:39.809176 (+ 130us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:39.809183 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:39.809224 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:39.811692 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:39.812360 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:39.812363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:39.812365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:39.812468 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:39.812733 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:39.812735 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:39.812736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:39.813184 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:39.813303 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:39.813621 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 18:32:39.813763 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.207.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.67.keytab.
Entry for principal oryx/127.28.207.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.67.keytab.
I20260929 18:32:39.834532 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.67:0
--local_ip_for_outbound_sockets=127.28.207.67
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:37575
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.67.keytab
--principal=oryx/127.28.207.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:45783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:39.953670 4917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:39.953963 4917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:39.954037 4917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:39.958213 4917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:39.958299 4917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:39.958371 4917 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 18:32:39.958449 4917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.67
I20260929 18:32:39.963826 4917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:45783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.67.keytab
--principal=oryx/127.28.207.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.207.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.28.207.126:37575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.67
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:39.965013 4917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:39.966127 4917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:39.977084 4925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:39.977119 4922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:39.977084 4923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:39.977991 4917 server_base.cc:1034] running on GCE node
I20260929 18:32:39.978447 4917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:39.979204 4917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:39.980418 4917 hybrid_clock.cc:648] HybridClock initialized: now 1790706759980389 us; error 51 us; skew 500 ppm
Sep 29 18:32:39 dist-test-slave-qcpv krb5kdc[4520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706759, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.207.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:39.983692 4917 init.cc:377] Logged in from keytab as oryx/127.28.207.67@KRBTEST.COM (short username oryx)
I20260929 18:32:39.984655 4917 webserver.cc:460] Webserver started at http://127.28.207.67:39469/ using document root <none> and password file <none>
I20260929 18:32:39.985252 4917 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:39.985344 4917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:39.985574 4917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:39.987746 4917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-2/data/instance:
uuid: "571b1f82c62640baa5f92127434ff04e"
format_stamp: "Formatted at 2026-09-29 18:32:39 on dist-test-slave-qcpv"
server_key: "f1a7741e440b2362c58d61a41029b6d3"
server_key_iv: "2acf7020b295619bfd2942c9debeb878"
server_key_version: "encryptionkey@0"
I20260929 18:32:39.988332 4917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance:
uuid: "571b1f82c62640baa5f92127434ff04e"
format_stamp: "Formatted at 2026-09-29 18:32:39 on dist-test-slave-qcpv"
server_key: "f1a7741e440b2362c58d61a41029b6d3"
server_key_iv: "2acf7020b295619bfd2942c9debeb878"
server_key_version: "encryptionkey@0"
I20260929 18:32:39.992578 4917 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:32:39.995564 4932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:39.997175 4917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:32:39.997357 4917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-2/wal
uuid: "571b1f82c62640baa5f92127434ff04e"
format_stamp: "Formatted at 2026-09-29 18:32:39 on dist-test-slave-qcpv"
server_key: "f1a7741e440b2362c58d61a41029b6d3"
server_key_iv: "2acf7020b295619bfd2942c9debeb878"
server_key_version: "encryptionkey@0"
I20260929 18:32:39.997484 4917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:40.026342 4917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:40.030365 4917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:40.030625 4917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:40.032413 4917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:40.032490 4917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:40.032572 4917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:40.032619 4917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 18:32:40 dist-test-slave-qcpv krb5kdc[4520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706759, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.207.67@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM
I20260929 18:32:40.046020 4917 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.67:40275
I20260929 18:32:40.046067 5047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.67:40275 every 8 connection(s)
I20260929 18:32:40.047236 4917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
I20260929 18:32:40.050818 4614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:40.033745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:36983 (local address 127.28.207.126:37575)
0929 18:32:40.033947 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:40.033954 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:40.034988 (+ 1034us) server_negotiation.cc:407] Connection header received
0929 18:32:40.036848 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:40.036853 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:40.036956 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:40.037069 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:40.038956 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.039566 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.040895 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.041213 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.041328 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:40.045495 (+ 4167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:40.045504 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:40.045509 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:40.045550 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:40.048129 (+ 2579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:40.048745 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:40.048748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:40.048750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:40.048915 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:40.049518 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:40.049522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:40.049523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:40.049797 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:40.049896 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:40.050497 (+ 601us) server_negotiation.cc:299] Negotiation successful
0929 18:32:40.050657 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 18:32:40.051829 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:40.034212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:37575 (local address 127.28.207.67:36983)
0929 18:32:40.034770 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:40.034829 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:40.036491 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:40.037309 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:40.037318 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:40.037992 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:40.038736 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:40.038763 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.039708 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.039712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:40.040758 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:40.040767 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.041402 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.041406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:40.041661 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:40.042480 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:40.042503 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:40.045112 (+ 2609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:40.048291 (+ 3179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:40.048297 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:40.048309 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:40.048626 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:40.049186 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:40.049189 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:40.049191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:40.049358 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:40.049911 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:40.049917 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:40.050207 (+ 290us) client_negotiation.cc:769] Sending connection context
0929 18:32:40.050384 (+ 177us) client_negotiation.cc:240] Negotiation successful
0929 18:32:40.050710 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":172,"threads_started":1}
I20260929 18:32:40.052834 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 4917
I20260929 18:32:40.052986 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance
I20260929 18:32:40.053284 29501 external_mini_cluster.cc:1428] Setting key db8d5e346e210948efa74b8e3a039cf9
I20260929 18:32:40.053719 5048 heartbeater.cc:347] Connected to a master server at 127.28.207.126:37575
I20260929 18:32:40.053920 5048 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:40.054582 5048 heartbeater.cc:511] Master 127.28.207.126:37575 requested a full tablet report, sending...
I20260929 18:32:40.055980 4565 ts_manager.cc:194] Registered new tserver with Master: 571b1f82c62640baa5f92127434ff04e (127.28.207.67:40275)
I20260929 18:32:40.057017 4565 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.28.207.67@KRBTEST.COM'} at 127.28.207.67:36983
I20260929 18:32:40.065223 4614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:40.057036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:60259 (local address 127.28.207.126:37575)
0929 18:32:40.057211 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:40.057218 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:40.057445 (+ 227us) server_negotiation.cc:407] Connection header received
0929 18:32:40.057593 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:40.057598 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:40.057662 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:40.057788 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:40.058376 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.059048 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.059951 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.060201 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.060318 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:40.061590 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:40.061594 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:40.061597 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:40.061625 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:40.063502 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:40.063941 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:40.063945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:40.063947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:40.064033 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:40.064302 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:40.064310 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:40.064312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:40.064512 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:40.064616 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:40.064895 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 18:32:40.065040 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 18:32:40.065238 5054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:40.056899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:37575 (local address 127.28.207.67:60259)
0929 18:32:40.057321 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:40.057343 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:40.057470 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:40.057817 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:40.057822 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:40.058081 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:40.058238 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:40.058246 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.059181 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.059185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:40.059810 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:40.059819 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.060321 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.060325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:40.060422 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:40.061061 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:40.061074 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:40.061424 (+ 350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:40.063627 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:40.063632 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:40.063635 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:40.063839 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:40.064130 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:40.064138 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:40.064140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:40.064214 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:40.064643 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:40.064646 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:40.064737 (+ 91us) client_negotiation.cc:769] Sending connection context
0929 18:32:40.064842 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 18:32:40.065042 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":1,"thread_start_us":196,"threads_started":1}
I20260929 18:32:40.069749 29501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 18:32:40 dist-test-slave-qcpv krb5kdc[4520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706760, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 18:32:40 dist-test-slave-qcpv krb5kdc[4520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:40 dist-test-slave-qcpv krb5kdc[4520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 18:32:40.098438 4614 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:40.091102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32952 (local address 127.28.207.126:37575)
0929 18:32:40.091278 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:40.091282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:40.091300 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:32:40.091358 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:40.091360 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:40.091425 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:40.091514 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:40.092316 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.092974 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.093939 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.094175 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.094280 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:40.098204 (+ 3924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32952: BlockingRecv error: recv got EOF from 127.0.0.1:32952 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 18:32:40 dist-test-slave-qcpv krb5kdc[4520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706760, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM
I20260929 18:32:40.109575 4614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:40.099962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32968 (local address 127.28.207.126:37575)
0929 18:32:40.100129 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:40.100134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:40.100238 (+ 104us) server_negotiation.cc:407] Connection header received
0929 18:32:40.100465 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:40.100467 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:40.100516 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:40.100606 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:40.101358 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.102025 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.103087 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.103304 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.103388 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:40.106214 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:40.106217 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:40.106219 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:40.106243 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:40.107782 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:40.108303 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:40.108306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:40.108307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:40.108383 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:40.108711 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:40.108713 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:40.108714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:40.108887 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:40.109013 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:40.109300 (+ 287us) server_negotiation.cc:299] Negotiation successful
0929 18:32:40.109398 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 18:32:40.114202 4565 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:32:40.117507 4565 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:32:40.136521 5074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:40.128672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:38545 (local address 127.0.0.1:33144)
0929 18:32:40.130224 (+ 1552us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:40.130238 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:40.130345 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:40.130817 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:40.130820 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:40.130837 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:40.131056 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:40.131063 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.132644 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.132648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:40.134272 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:40.134285 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.135812 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.135816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:40.135951 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:40.135999 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 18:32:40.136128 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 18:32:40.136297 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1487,"mutex_wait_us":167,"thread_start_us":370,"threads_started":1}
I20260929 18:32:40.137197 5073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:40.129443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33144 (local address 127.28.207.66:38545)
0929 18:32:40.130244 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:40.130256 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:40.130320 (+ 64us) server_negotiation.cc:407] Connection header received
0929 18:32:40.130485 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:40.130491 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:40.130665 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:40.130824 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:40.131200 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.132487 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.134431 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.135396 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.135515 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:40.135731 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:40.136878 (+ 1147us) server_negotiation.cc:299] Negotiation successful
0929 18:32:40.137010 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":148,"threads_started":1}
I20260929 18:32:40.137535 5071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:40.128694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:40275 (local address 127.0.0.1:33892)
0929 18:32:40.129951 (+ 1257us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:40.129968 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:40.130075 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:40.131077 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:40.131080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:40.131100 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:40.131214 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:40.131221 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.132644 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.132648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:40.134272 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:40.134285 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.135975 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.135984 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:40.136236 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:40.136312 (+ 76us) client_negotiation.cc:769] Sending connection context
0929 18:32:40.136556 (+ 244us) client_negotiation.cc:240] Negotiation successful
0929 18:32:40.136920 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1174,"mutex_wait_us":125,"thread_start_us":994,"threads_started":1}
I20260929 18:32:40.137862 5075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:40.129792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33892 (local address 127.28.207.67:40275)
0929 18:32:40.130412 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:40.130456 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:40.130486 (+ 30us) server_negotiation.cc:407] Connection header received
0929 18:32:40.130588 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:40.130594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:40.130900 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:40.131089 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:40.131340 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.132487 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.134431 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.135396 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.135515 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:40.135857 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:40.136669 (+ 812us) server_negotiation.cc:299] Negotiation successful
0929 18:32:40.137180 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":119,"threads_started":1}
I20260929 18:32:40.141031 4842 tablet_service.cc:1467] Processing CreateTablet for tablet a94925a92dae420a92aaddab8e360a13 (DEFAULT_TABLE table=test-table [id=57df7b927c55417aa8cb0463a24ebc48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:40.141132 4982 tablet_service.cc:1467] Processing CreateTablet for tablet a94925a92dae420a92aaddab8e360a13 (DEFAULT_TABLE table=test-table [id=57df7b927c55417aa8cb0463a24ebc48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:40.142202 4982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a94925a92dae420a92aaddab8e360a13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:40.142202 4842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a94925a92dae420a92aaddab8e360a13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:40.142585 5072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:40.129454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40364 (local address 127.28.207.65:38705)
0929 18:32:40.129710 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:40.129717 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:40.129740 (+ 23us) server_negotiation.cc:407] Connection header received
0929 18:32:40.129816 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:40.129822 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:40.130046 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:40.130614 (+ 568us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:40.130740 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.131698 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.139171 (+ 7473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.141729 (+ 2558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.141898 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:40.142096 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:40.142178 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 18:32:40.142308 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":105,"threads_started":1}
I20260929 18:32:40.142645 5070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:40.128517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:38705 (local address 127.0.0.1:40364)
0929 18:32:40.129049 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:40.129101 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:40.129236 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:40.130226 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:40.130233 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:40.130261 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:40.130460 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:40.130469 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.131800 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.131805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:40.138946 (+ 7141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:40.138963 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.141907 (+ 2944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.141911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:40.142028 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:40.142055 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 18:32:40.142130 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 18:32:40.142239 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":100,"threads_started":1}
I20260929 18:32:40.146379 4702 tablet_service.cc:1467] Processing CreateTablet for tablet a94925a92dae420a92aaddab8e360a13 (DEFAULT_TABLE table=test-table [id=57df7b927c55417aa8cb0463a24ebc48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:40.147675 4702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a94925a92dae420a92aaddab8e360a13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:40.152370 5076 tablet_bootstrap.cc:492] T a94925a92dae420a92aaddab8e360a13 P 4ee346edbe3a4443bb23cccccd8b9f75: Bootstrap starting.
I20260929 18:32:40.155020 5077 tablet_bootstrap.cc:492] T a94925a92dae420a92aaddab8e360a13 P 571b1f82c62640baa5f92127434ff04e: Bootstrap starting.
I20260929 18:32:40.155447 5076 tablet_bootstrap.cc:654] T a94925a92dae420a92aaddab8e360a13 P 4ee346edbe3a4443bb23cccccd8b9f75: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:40.156344 5078 tablet_bootstrap.cc:492] T a94925a92dae420a92aaddab8e360a13 P 4eb628a7dcc2441aa375002c139dcace: Bootstrap starting.
I20260929 18:32:40.156478 5076 log.cc:824] T a94925a92dae420a92aaddab8e360a13 P 4ee346edbe3a4443bb23cccccd8b9f75: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:40.157636 5077 tablet_bootstrap.cc:654] T a94925a92dae420a92aaddab8e360a13 P 571b1f82c62640baa5f92127434ff04e: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:40.158563 5077 log.cc:824] T a94925a92dae420a92aaddab8e360a13 P 571b1f82c62640baa5f92127434ff04e: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:40.158630 5078 tablet_bootstrap.cc:654] T a94925a92dae420a92aaddab8e360a13 P 4eb628a7dcc2441aa375002c139dcace: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:40.158859 5076 tablet_bootstrap.cc:492] T a94925a92dae420a92aaddab8e360a13 P 4ee346edbe3a4443bb23cccccd8b9f75: No bootstrap required, opened a new log
I20260929 18:32:40.159139 5076 ts_tablet_manager.cc:1397] T a94925a92dae420a92aaddab8e360a13 P 4ee346edbe3a4443bb23cccccd8b9f75: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.002s
I20260929 18:32:40.159451 5078 log.cc:824] T a94925a92dae420a92aaddab8e360a13 P 4eb628a7dcc2441aa375002c139dcace: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:40.160897 5077 tablet_bootstrap.cc:492] T a94925a92dae420a92aaddab8e360a13 P 571b1f82c62640baa5f92127434ff04e: No bootstrap required, opened a new log
I20260929 18:32:40.161118 5077 ts_tablet_manager.cc:1397] T a94925a92dae420a92aaddab8e360a13 P 571b1f82c62640baa5f92127434ff04e: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260929 18:32:40.161607 5078 tablet_bootstrap.cc:492] T a94925a92dae420a92aaddab8e360a13 P 4eb628a7dcc2441aa375002c139dcace: No bootstrap required, opened a new log
I20260929 18:32:40.161788 5078 ts_tablet_manager.cc:1397] T a94925a92dae420a92aaddab8e360a13 P 4eb628a7dcc2441aa375002c139dcace: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:32:40.163096 5076 raft_consensus.cc:348] T a94925a92dae420a92aaddab8e360a13 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:40.163381 5076 raft_consensus.cc:374] T a94925a92dae420a92aaddab8e360a13 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:40.163435 5076 raft_consensus.cc:729] T a94925a92dae420a92aaddab8e360a13 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ee346edbe3a4443bb23cccccd8b9f75, State: Initialized, Role: FOLLOWER
I20260929 18:32:40.163923 5076 consensus_queue.cc:260] T a94925a92dae420a92aaddab8e360a13 P 4ee346edbe3a4443bb23cccccd8b9f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:40.164763 4908 heartbeater.cc:503] Master 127.28.207.126:37575 was elected leader, sending a full tablet report...
I20260929 18:32:40.165160 5076 ts_tablet_manager.cc:1428] T a94925a92dae420a92aaddab8e360a13 P 4ee346edbe3a4443bb23cccccd8b9f75: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260929 18:32:40.165627 5077 raft_consensus.cc:348] T a94925a92dae420a92aaddab8e360a13 P 571b1f82c62640baa5f92127434ff04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:40.165956 5077 raft_consensus.cc:374] T a94925a92dae420a92aaddab8e360a13 P 571b1f82c62640baa5f92127434ff04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:40.166035 5077 raft_consensus.cc:729] T a94925a92dae420a92aaddab8e360a13 P 571b1f82c62640baa5f92127434ff04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 571b1f82c62640baa5f92127434ff04e, State: Initialized, Role: FOLLOWER
I20260929 18:32:40.166409 5078 raft_consensus.cc:348] T a94925a92dae420a92aaddab8e360a13 P 4eb628a7dcc2441aa375002c139dcace [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:40.166563 5077 consensus_queue.cc:260] T a94925a92dae420a92aaddab8e360a13 P 571b1f82c62640baa5f92127434ff04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:40.166652 5078 raft_consensus.cc:374] T a94925a92dae420a92aaddab8e360a13 P 4eb628a7dcc2441aa375002c139dcace [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:40.166699 5078 raft_consensus.cc:729] T a94925a92dae420a92aaddab8e360a13 P 4eb628a7dcc2441aa375002c139dcace [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb628a7dcc2441aa375002c139dcace, State: Initialized, Role: FOLLOWER
I20260929 18:32:40.167155 5078 consensus_queue.cc:260] T a94925a92dae420a92aaddab8e360a13 P 4eb628a7dcc2441aa375002c139dcace [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:40.167689 5077 ts_tablet_manager.cc:1428] T a94925a92dae420a92aaddab8e360a13 P 571b1f82c62640baa5f92127434ff04e: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260929 18:32:40.167943 4768 heartbeater.cc:503] Master 127.28.207.126:37575 was elected leader, sending a full tablet report...
I20260929 18:32:40.168057 5048 heartbeater.cc:503] Master 127.28.207.126:37575 was elected leader, sending a full tablet report...
I20260929 18:32:40.168861 5078 ts_tablet_manager.cc:1428] T a94925a92dae420a92aaddab8e360a13 P 4eb628a7dcc2441aa375002c139dcace: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260929 18:32:40.256438 5082 raft_consensus.cc:482] T a94925a92dae420a92aaddab8e360a13 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:32:40.256729 5082 raft_consensus.cc:504] T a94925a92dae420a92aaddab8e360a13 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:40.258315 5082 leader_election.cc:288] T a94925a92dae420a92aaddab8e360a13 P 4ee346edbe3a4443bb23cccccd8b9f75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4eb628a7dcc2441aa375002c139dcace (127.28.207.65:38705), 571b1f82c62640baa5f92127434ff04e (127.28.207.67:40275)
I20260929 18:32:40.263407 5072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:40.259110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:50903 (local address 127.28.207.65:38705)
0929 18:32:40.259267 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:40.259272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:40.259292 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:32:40.259364 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:40.259367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:40.259428 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:40.259551 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:40.259869 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.260599 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.261878 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.262580 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.262726 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:40.262790 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:40.263173 (+ 383us) server_negotiation.cc:299] Negotiation successful
0929 18:32:40.263274 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:32:40.263424 4801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:40.258641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:38705 (local address 127.28.207.66:50903)
0929 18:32:40.258757 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:40.258773 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:40.258947 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:40.259573 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:40.259578 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:40.259599 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:40.259736 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:40.259743 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.260748 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.260776 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:40.261703 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:40.261713 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.262766 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.262769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:40.262993 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:40.263027 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 18:32:40.263130 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 18:32:40.263232 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":1}
I20260929 18:32:40.264232 5075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:40.258787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:32821 (local address 127.28.207.67:40275)
0929 18:32:40.259041 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:40.259047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:40.259342 (+ 295us) server_negotiation.cc:407] Connection header received
0929 18:32:40.259465 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:40.259468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:40.259540 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:40.259622 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:40.260106 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.260732 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.263008 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.263642 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.263776 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:40.263834 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:40.264045 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 18:32:40.264116 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260929 18:32:40.264256 5085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:40.258641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:40275 (local address 127.28.207.66:32821)
0929 18:32:40.259236 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:40.259248 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:40.259372 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:40.259723 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:40.259725 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:40.259747 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:40.259926 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:40.259935 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.260933 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.260937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:40.262850 (+ 1913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:40.262892 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.263802 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.263805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:40.263918 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:40.263942 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 18:32:40.264030 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 18:32:40.264112 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":129,"thread_start_us":312,"threads_started":1}
I20260929 18:32:40.264259 4722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94925a92dae420a92aaddab8e360a13" candidate_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb628a7dcc2441aa375002c139dcace" is_pre_election: true
I20260929 18:32:40.264756 4722 raft_consensus.cc:2454] T a94925a92dae420a92aaddab8e360a13 P 4eb628a7dcc2441aa375002c139dcace [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ee346edbe3a4443bb23cccccd8b9f75 in term 0.
I20260929 18:32:40.265245 5002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94925a92dae420a92aaddab8e360a13" candidate_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "571b1f82c62640baa5f92127434ff04e" is_pre_election: true
I20260929 18:32:40.265427 4793 leader_election.cc:302] T a94925a92dae420a92aaddab8e360a13 P 4ee346edbe3a4443bb23cccccd8b9f75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb628a7dcc2441aa375002c139dcace, 4ee346edbe3a4443bb23cccccd8b9f75; no voters:
I20260929 18:32:40.265553 5002 raft_consensus.cc:2454] T a94925a92dae420a92aaddab8e360a13 P 571b1f82c62640baa5f92127434ff04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ee346edbe3a4443bb23cccccd8b9f75 in term 0.
I20260929 18:32:40.265848 5082 raft_consensus.cc:2781] T a94925a92dae420a92aaddab8e360a13 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:32:40.265985 5082 raft_consensus.cc:482] T a94925a92dae420a92aaddab8e360a13 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:32:40.266055 5082 raft_consensus.cc:3037] T a94925a92dae420a92aaddab8e360a13 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:40.267735 5082 raft_consensus.cc:504] T a94925a92dae420a92aaddab8e360a13 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:40.268173 5082 leader_election.cc:288] T a94925a92dae420a92aaddab8e360a13 P 4ee346edbe3a4443bb23cccccd8b9f75 [CANDIDATE]: Term 1 election: Requested vote from peers 4eb628a7dcc2441aa375002c139dcace (127.28.207.65:38705), 571b1f82c62640baa5f92127434ff04e (127.28.207.67:40275)
I20260929 18:32:40.268784 4722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94925a92dae420a92aaddab8e360a13" candidate_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb628a7dcc2441aa375002c139dcace"
I20260929 18:32:40.268784 5002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a94925a92dae420a92aaddab8e360a13" candidate_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "571b1f82c62640baa5f92127434ff04e"
I20260929 18:32:40.268983 5002 raft_consensus.cc:3037] T a94925a92dae420a92aaddab8e360a13 P 571b1f82c62640baa5f92127434ff04e [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:40.268983 4722 raft_consensus.cc:3037] T a94925a92dae420a92aaddab8e360a13 P 4eb628a7dcc2441aa375002c139dcace [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:40.270426 5002 raft_consensus.cc:2454] T a94925a92dae420a92aaddab8e360a13 P 571b1f82c62640baa5f92127434ff04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ee346edbe3a4443bb23cccccd8b9f75 in term 1.
I20260929 18:32:40.270427 4722 raft_consensus.cc:2454] T a94925a92dae420a92aaddab8e360a13 P 4eb628a7dcc2441aa375002c139dcace [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ee346edbe3a4443bb23cccccd8b9f75 in term 1.
I20260929 18:32:40.270834 4793 leader_election.cc:302] T a94925a92dae420a92aaddab8e360a13 P 4ee346edbe3a4443bb23cccccd8b9f75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb628a7dcc2441aa375002c139dcace, 4ee346edbe3a4443bb23cccccd8b9f75; no voters:
I20260929 18:32:40.271090 5082 raft_consensus.cc:2781] T a94925a92dae420a92aaddab8e360a13 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:32:40.271337 5082 raft_consensus.cc:686] T a94925a92dae420a92aaddab8e360a13 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 1 LEADER]: Becoming Leader. State: Replica: 4ee346edbe3a4443bb23cccccd8b9f75, State: Running, Role: LEADER
I20260929 18:32:40.271682 5082 consensus_queue.cc:237] T a94925a92dae420a92aaddab8e360a13 P 4ee346edbe3a4443bb23cccccd8b9f75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:40.275349 4562 catalog_manager.cc:5495] T a94925a92dae420a92aaddab8e360a13 P 4ee346edbe3a4443bb23cccccd8b9f75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ee346edbe3a4443bb23cccccd8b9f75 (127.28.207.66). New cstate: current_term: 1 leader_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } health_report { overall_health: UNKNOWN } } }
W20260929 18:32:40.295977 4909 tablet.cc:2369] T a94925a92dae420a92aaddab8e360a13 P 4ee346edbe3a4443bb23cccccd8b9f75: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:32:40.296468 5073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:40.291636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33148 (local address 127.28.207.66:38545)
0929 18:32:40.291779 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:40.291783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:40.291800 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:32:40.291899 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:40.291901 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:40.291964 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:40.292099 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:32:40.292485 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.293251 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.294411 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.294678 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.294781 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:40.295144 (+ 363us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:32:40.295962 (+ 818us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:32:40.296119 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:40.296222 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 18:32:40.296301 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 18:32:40.298501 5049 tablet.cc:2369] T a94925a92dae420a92aaddab8e360a13 P 571b1f82c62640baa5f92127434ff04e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:32:40.299840 4769 tablet.cc:2369] T a94925a92dae420a92aaddab8e360a13 P 4eb628a7dcc2441aa375002c139dcace: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:32:40.304023 4722 raft_consensus.cc:1261] T a94925a92dae420a92aaddab8e360a13 P 4eb628a7dcc2441aa375002c139dcace [term 1 FOLLOWER]: Refusing update from remote peer 4ee346edbe3a4443bb23cccccd8b9f75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:32:40.304023 5002 raft_consensus.cc:1261] T a94925a92dae420a92aaddab8e360a13 P 571b1f82c62640baa5f92127434ff04e [term 1 FOLLOWER]: Refusing update from remote peer 4ee346edbe3a4443bb23cccccd8b9f75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:32:40.305402 5086 consensus_queue.cc:1035] T a94925a92dae420a92aaddab8e360a13 P 4ee346edbe3a4443bb23cccccd8b9f75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:32:40.305632 5082 consensus_queue.cc:1035] T a94925a92dae420a92aaddab8e360a13 P 4ee346edbe3a4443bb23cccccd8b9f75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:32:40.318063 5091 mvcc.cc:205] Tried to move back new op lower bound from 7334734890196439040 to 7334734890075766784. Current Snapshot: MvccSnapshot[applied={T|T < 7334734890196439040}]
I20260929 18:32:40.326448 4565 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:32:40.326609 4565 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:32:40.329727 4565 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf94f8dce239472093ae97936b750185: Sending DeleteTablet for 3 replicas of tablet a94925a92dae420a92aaddab8e360a13
I20260929 18:32:40.330551 4982 tablet_service.cc:1514] Processing DeleteTablet for tablet a94925a92dae420a92aaddab8e360a13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:40 UTC) from {username='oryx', principal='oryx/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:33892
I20260929 18:32:40.330549 4702 tablet_service.cc:1514] Processing DeleteTablet for tablet a94925a92dae420a92aaddab8e360a13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:40 UTC) from {username='oryx', principal='oryx/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:40364
I20260929 18:32:40.331118 4842 tablet_service.cc:1514] Processing DeleteTablet for tablet a94925a92dae420a92aaddab8e360a13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:40 UTC) from {username='oryx', principal='oryx/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:33144
I20260929 18:32:40.331274 5100 tablet_replica.cc:330] T a94925a92dae420a92aaddab8e360a13 P 571b1f82c62640baa5f92127434ff04e: stopping tablet replica
I20260929 18:32:40.331584 5100 raft_consensus.cc:2229] T a94925a92dae420a92aaddab8e360a13 P 571b1f82c62640baa5f92127434ff04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:32:40.331724 5101 tablet_replica.cc:330] T a94925a92dae420a92aaddab8e360a13 P 4ee346edbe3a4443bb23cccccd8b9f75: stopping tablet replica
I20260929 18:32:40.331946 5100 raft_consensus.cc:2258] T a94925a92dae420a92aaddab8e360a13 P 571b1f82c62640baa5f92127434ff04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:40.332321 5099 tablet_replica.cc:330] T a94925a92dae420a92aaddab8e360a13 P 4eb628a7dcc2441aa375002c139dcace: stopping tablet replica
I20260929 18:32:40.332333 4565 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 18:32:40.332587 5099 raft_consensus.cc:2229] T a94925a92dae420a92aaddab8e360a13 P 4eb628a7dcc2441aa375002c139dcace [term 1 FOLLOWER]: Raft consensus shutting down.
W20260929 18:32:40.332783 4565 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:32:40.332774 5101 raft_consensus.cc:2229] T a94925a92dae420a92aaddab8e360a13 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:32:40.332851 5099 raft_consensus.cc:2258] T a94925a92dae420a92aaddab8e360a13 P 4eb628a7dcc2441aa375002c139dcace [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:40.333598 5101 raft_consensus.cc:2258] T a94925a92dae420a92aaddab8e360a13 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:40.334156 5100 ts_tablet_manager.cc:1905] T a94925a92dae420a92aaddab8e360a13 P 571b1f82c62640baa5f92127434ff04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:32:40.336571 5099 ts_tablet_manager.cc:1905] T a94925a92dae420a92aaddab8e360a13 P 4eb628a7dcc2441aa375002c139dcace: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:32:40.338830 5100 ts_tablet_manager.cc:1918] T a94925a92dae420a92aaddab8e360a13 P 571b1f82c62640baa5f92127434ff04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:32:40.338999 5100 log.cc:1196] T a94925a92dae420a92aaddab8e360a13 P 571b1f82c62640baa5f92127434ff04e: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-2/wal/wals/a94925a92dae420a92aaddab8e360a13
I20260929 18:32:40.339108 4702 tablet_service.cc:1467] Processing CreateTablet for tablet f173539a9a314366938e389af35dcd80 (DEFAULT_TABLE table=test-table [id=d95aa78406654e94a91cbc942c0ed3f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:40.339362 5100 ts_tablet_manager.cc:1939] T a94925a92dae420a92aaddab8e360a13 P 571b1f82c62640baa5f92127434ff04e: Deleting consensus metadata
I20260929 18:32:40.339407 4702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f173539a9a314366938e389af35dcd80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:40.340071 5101 ts_tablet_manager.cc:1905] T a94925a92dae420a92aaddab8e360a13 P 4ee346edbe3a4443bb23cccccd8b9f75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:32:40.341930 5078 tablet_bootstrap.cc:492] T f173539a9a314366938e389af35dcd80 P 4eb628a7dcc2441aa375002c139dcace: Bootstrap starting.
I20260929 18:32:40.342300 5099 ts_tablet_manager.cc:1918] T a94925a92dae420a92aaddab8e360a13 P 4eb628a7dcc2441aa375002c139dcace: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:32:40.342437 5099 log.cc:1196] T a94925a92dae420a92aaddab8e360a13 P 4eb628a7dcc2441aa375002c139dcace: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-0/wal/wals/a94925a92dae420a92aaddab8e360a13
I20260929 18:32:40.342808 5099 ts_tablet_manager.cc:1939] T a94925a92dae420a92aaddab8e360a13 P 4eb628a7dcc2441aa375002c139dcace: Deleting consensus metadata
I20260929 18:32:40.343022 5078 tablet_bootstrap.cc:654] T f173539a9a314366938e389af35dcd80 P 4eb628a7dcc2441aa375002c139dcace: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:40.343849 5101 ts_tablet_manager.cc:1918] T a94925a92dae420a92aaddab8e360a13 P 4ee346edbe3a4443bb23cccccd8b9f75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:32:40.344024 5101 log.cc:1196] T a94925a92dae420a92aaddab8e360a13 P 4ee346edbe3a4443bb23cccccd8b9f75: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-1/wal/wals/a94925a92dae420a92aaddab8e360a13
I20260929 18:32:40.344687 4549 catalog_manager.cc:4841] TS 4eb628a7dcc2441aa375002c139dcace (127.28.207.65:38705): tablet a94925a92dae420a92aaddab8e360a13 (table test-table [id=57df7b927c55417aa8cb0463a24ebc48]) successfully deleted
I20260929 18:32:40.344769 5101 ts_tablet_manager.cc:1939] T a94925a92dae420a92aaddab8e360a13 P 4ee346edbe3a4443bb23cccccd8b9f75: Deleting consensus metadata
I20260929 18:32:40.344751 4552 catalog_manager.cc:4841] TS 571b1f82c62640baa5f92127434ff04e (127.28.207.67:40275): tablet a94925a92dae420a92aaddab8e360a13 (table test-table [id=57df7b927c55417aa8cb0463a24ebc48]) successfully deleted
I20260929 18:32:40.345348 4842 tablet_service.cc:1467] Processing CreateTablet for tablet f173539a9a314366938e389af35dcd80 (DEFAULT_TABLE table=test-table [id=d95aa78406654e94a91cbc942c0ed3f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:40.345402 4982 tablet_service.cc:1467] Processing CreateTablet for tablet f173539a9a314366938e389af35dcd80 (DEFAULT_TABLE table=test-table [id=d95aa78406654e94a91cbc942c0ed3f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:40.345681 4842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f173539a9a314366938e389af35dcd80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:40.345721 4982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f173539a9a314366938e389af35dcd80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:40.347833 5076 tablet_bootstrap.cc:492] T f173539a9a314366938e389af35dcd80 P 4ee346edbe3a4443bb23cccccd8b9f75: Bootstrap starting.
I20260929 18:32:40.348410 4550 catalog_manager.cc:4841] TS 4ee346edbe3a4443bb23cccccd8b9f75 (127.28.207.66:38545): tablet a94925a92dae420a92aaddab8e360a13 (table test-table [id=57df7b927c55417aa8cb0463a24ebc48]) successfully deleted
I20260929 18:32:40.349135 5076 tablet_bootstrap.cc:654] T f173539a9a314366938e389af35dcd80 P 4ee346edbe3a4443bb23cccccd8b9f75: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:40.349431 5078 tablet_bootstrap.cc:492] T f173539a9a314366938e389af35dcd80 P 4eb628a7dcc2441aa375002c139dcace: No bootstrap required, opened a new log
I20260929 18:32:40.349565 5078 ts_tablet_manager.cc:1397] T f173539a9a314366938e389af35dcd80 P 4eb628a7dcc2441aa375002c139dcace: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20260929 18:32:40.350117 5078 raft_consensus.cc:348] T f173539a9a314366938e389af35dcd80 P 4eb628a7dcc2441aa375002c139dcace [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:40.350270 5078 raft_consensus.cc:374] T f173539a9a314366938e389af35dcd80 P 4eb628a7dcc2441aa375002c139dcace [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:40.350306 5078 raft_consensus.cc:729] T f173539a9a314366938e389af35dcd80 P 4eb628a7dcc2441aa375002c139dcace [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb628a7dcc2441aa375002c139dcace, State: Initialized, Role: FOLLOWER
I20260929 18:32:40.350493 5078 consensus_queue.cc:260] T f173539a9a314366938e389af35dcd80 P 4eb628a7dcc2441aa375002c139dcace [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:40.350718 5076 tablet_bootstrap.cc:492] T f173539a9a314366938e389af35dcd80 P 4ee346edbe3a4443bb23cccccd8b9f75: No bootstrap required, opened a new log
I20260929 18:32:40.350793 5076 ts_tablet_manager.cc:1397] T f173539a9a314366938e389af35dcd80 P 4ee346edbe3a4443bb23cccccd8b9f75: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260929 18:32:40.350785 5078 ts_tablet_manager.cc:1428] T f173539a9a314366938e389af35dcd80 P 4eb628a7dcc2441aa375002c139dcace: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 18:32:40.351408 5077 tablet_bootstrap.cc:492] T f173539a9a314366938e389af35dcd80 P 571b1f82c62640baa5f92127434ff04e: Bootstrap starting.
I20260929 18:32:40.351361 5076 raft_consensus.cc:348] T f173539a9a314366938e389af35dcd80 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:40.351476 5076 raft_consensus.cc:374] T f173539a9a314366938e389af35dcd80 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:40.351506 5076 raft_consensus.cc:729] T f173539a9a314366938e389af35dcd80 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ee346edbe3a4443bb23cccccd8b9f75, State: Initialized, Role: FOLLOWER
I20260929 18:32:40.351665 5076 consensus_queue.cc:260] T f173539a9a314366938e389af35dcd80 P 4ee346edbe3a4443bb23cccccd8b9f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:40.352438 5076 ts_tablet_manager.cc:1428] T f173539a9a314366938e389af35dcd80 P 4ee346edbe3a4443bb23cccccd8b9f75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:32:40.352630 5077 tablet_bootstrap.cc:654] T f173539a9a314366938e389af35dcd80 P 571b1f82c62640baa5f92127434ff04e: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:40.353875 5077 tablet_bootstrap.cc:492] T f173539a9a314366938e389af35dcd80 P 571b1f82c62640baa5f92127434ff04e: No bootstrap required, opened a new log
I20260929 18:32:40.353976 5077 ts_tablet_manager.cc:1397] T f173539a9a314366938e389af35dcd80 P 571b1f82c62640baa5f92127434ff04e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 18:32:40.354465 5077 raft_consensus.cc:348] T f173539a9a314366938e389af35dcd80 P 571b1f82c62640baa5f92127434ff04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:40.354607 5077 raft_consensus.cc:374] T f173539a9a314366938e389af35dcd80 P 571b1f82c62640baa5f92127434ff04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:40.354650 5077 raft_consensus.cc:729] T f173539a9a314366938e389af35dcd80 P 571b1f82c62640baa5f92127434ff04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 571b1f82c62640baa5f92127434ff04e, State: Initialized, Role: FOLLOWER
I20260929 18:32:40.354823 5077 consensus_queue.cc:260] T f173539a9a314366938e389af35dcd80 P 571b1f82c62640baa5f92127434ff04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:40.355602 5077 ts_tablet_manager.cc:1428] T f173539a9a314366938e389af35dcd80 P 571b1f82c62640baa5f92127434ff04e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 18:32:40.429471 5086 raft_consensus.cc:482] T f173539a9a314366938e389af35dcd80 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:32:40.429678 5086 raft_consensus.cc:504] T f173539a9a314366938e389af35dcd80 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:40.430260 5086 leader_election.cc:288] T f173539a9a314366938e389af35dcd80 P 4ee346edbe3a4443bb23cccccd8b9f75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4eb628a7dcc2441aa375002c139dcace (127.28.207.65:38705), 571b1f82c62640baa5f92127434ff04e (127.28.207.67:40275)
I20260929 18:32:40.430953 4722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f173539a9a314366938e389af35dcd80" candidate_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb628a7dcc2441aa375002c139dcace" is_pre_election: true
I20260929 18:32:40.430989 5002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f173539a9a314366938e389af35dcd80" candidate_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "571b1f82c62640baa5f92127434ff04e" is_pre_election: true
I20260929 18:32:40.431143 4722 raft_consensus.cc:2454] T f173539a9a314366938e389af35dcd80 P 4eb628a7dcc2441aa375002c139dcace [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ee346edbe3a4443bb23cccccd8b9f75 in term 0.
I20260929 18:32:40.431231 5002 raft_consensus.cc:2454] T f173539a9a314366938e389af35dcd80 P 571b1f82c62640baa5f92127434ff04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ee346edbe3a4443bb23cccccd8b9f75 in term 0.
I20260929 18:32:40.431520 4793 leader_election.cc:302] T f173539a9a314366938e389af35dcd80 P 4ee346edbe3a4443bb23cccccd8b9f75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb628a7dcc2441aa375002c139dcace, 4ee346edbe3a4443bb23cccccd8b9f75; no voters:
I20260929 18:32:40.431761 5086 raft_consensus.cc:2781] T f173539a9a314366938e389af35dcd80 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:32:40.431855 5086 raft_consensus.cc:482] T f173539a9a314366938e389af35dcd80 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:32:40.431901 5086 raft_consensus.cc:3037] T f173539a9a314366938e389af35dcd80 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:40.433394 5086 raft_consensus.cc:504] T f173539a9a314366938e389af35dcd80 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:40.433907 5086 leader_election.cc:288] T f173539a9a314366938e389af35dcd80 P 4ee346edbe3a4443bb23cccccd8b9f75 [CANDIDATE]: Term 1 election: Requested vote from peers 4eb628a7dcc2441aa375002c139dcace (127.28.207.65:38705), 571b1f82c62640baa5f92127434ff04e (127.28.207.67:40275)
I20260929 18:32:40.434706 5002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f173539a9a314366938e389af35dcd80" candidate_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "571b1f82c62640baa5f92127434ff04e"
I20260929 18:32:40.434931 5002 raft_consensus.cc:3037] T f173539a9a314366938e389af35dcd80 P 571b1f82c62640baa5f92127434ff04e [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:40.434759 4722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f173539a9a314366938e389af35dcd80" candidate_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb628a7dcc2441aa375002c139dcace"
I20260929 18:32:40.435068 4722 raft_consensus.cc:3037] T f173539a9a314366938e389af35dcd80 P 4eb628a7dcc2441aa375002c139dcace [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:40.436396 5002 raft_consensus.cc:2454] T f173539a9a314366938e389af35dcd80 P 571b1f82c62640baa5f92127434ff04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ee346edbe3a4443bb23cccccd8b9f75 in term 1.
I20260929 18:32:40.436707 4722 raft_consensus.cc:2454] T f173539a9a314366938e389af35dcd80 P 4eb628a7dcc2441aa375002c139dcace [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ee346edbe3a4443bb23cccccd8b9f75 in term 1.
I20260929 18:32:40.436897 4796 leader_election.cc:302] T f173539a9a314366938e389af35dcd80 P 4ee346edbe3a4443bb23cccccd8b9f75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ee346edbe3a4443bb23cccccd8b9f75, 571b1f82c62640baa5f92127434ff04e; no voters:
I20260929 18:32:40.437141 5086 raft_consensus.cc:2781] T f173539a9a314366938e389af35dcd80 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:32:40.437300 5086 raft_consensus.cc:686] T f173539a9a314366938e389af35dcd80 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 1 LEADER]: Becoming Leader. State: Replica: 4ee346edbe3a4443bb23cccccd8b9f75, State: Running, Role: LEADER
I20260929 18:32:40.437474 5086 consensus_queue.cc:237] T f173539a9a314366938e389af35dcd80 P 4ee346edbe3a4443bb23cccccd8b9f75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:40.439630 4562 catalog_manager.cc:5495] T f173539a9a314366938e389af35dcd80 P 4ee346edbe3a4443bb23cccccd8b9f75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ee346edbe3a4443bb23cccccd8b9f75 (127.28.207.66). New cstate: current_term: 1 leader_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } health_report { overall_health: UNKNOWN } } }
Sep 29 18:32:40 dist-test-slave-qcpv krb5kdc[4520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706759, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.207.126@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM
I20260929 18:32:40.496616 5111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:40.485239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:37575 (local address 127.0.0.1:32980)
0929 18:32:40.485653 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:40.485672 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:40.485837 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:40.486213 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:40.486216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:40.486949 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:40.487129 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:40.487137 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.488011 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.488016 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:40.489005 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:40.489016 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.489617 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.489621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:40.489711 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:40.490493 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:40.490506 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:40.492210 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:40.494499 (+ 2289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:40.494510 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:40.494517 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:40.494950 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:40.495301 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:40.495305 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:40.495308 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:40.495421 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:40.495912 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:40.495919 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:40.496074 (+ 155us) client_negotiation.cc:769] Sending connection context
0929 18:32:40.496189 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 18:32:40.496351 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":137,"threads_started":1}
I20260929 18:32:40.496773 4614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:40.485341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32980 (local address 127.28.207.126:37575)
0929 18:32:40.485510 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:40.485516 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:40.485821 (+ 305us) server_negotiation.cc:407] Connection header received
0929 18:32:40.486011 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:40.486014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:40.486067 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:40.486160 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:40.487294 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.487868 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.489178 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.489440 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.489583 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:40.492378 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:40.492383 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:40.492387 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:40.492421 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:40.494336 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:40.495076 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:40.495080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:40.495082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:40.495178 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:40.495526 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:40.495529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:40.495531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:40.495767 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:40.495866 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:40.496292 (+ 426us) server_negotiation.cc:299] Negotiation successful
0929 18:32:40.496435 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 18:32:40.502233 4562 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:32980:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 18:32:40.502980 4562 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:32:40.507905 4702 tablet_service.cc:1467] Processing CreateTablet for tablet fb01ea03b01140bb96dc69b6bcf7bca9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a4525fee265c444a887c05b1ac4af001]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:32:40.508004 4982 tablet_service.cc:1467] Processing CreateTablet for tablet fb01ea03b01140bb96dc69b6bcf7bca9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a4525fee265c444a887c05b1ac4af001]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:32:40.508004 4842 tablet_service.cc:1467] Processing CreateTablet for tablet fb01ea03b01140bb96dc69b6bcf7bca9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a4525fee265c444a887c05b1ac4af001]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 18:32:40.508330 4702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb01ea03b01140bb96dc69b6bcf7bca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:40.508498 4842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb01ea03b01140bb96dc69b6bcf7bca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:40.508498 4982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb01ea03b01140bb96dc69b6bcf7bca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:40.511160 5078 tablet_bootstrap.cc:492] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4eb628a7dcc2441aa375002c139dcace: Bootstrap starting.
I20260929 18:32:40.511806 5076 tablet_bootstrap.cc:492] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4ee346edbe3a4443bb23cccccd8b9f75: Bootstrap starting.
I20260929 18:32:40.512493 5077 tablet_bootstrap.cc:492] T fb01ea03b01140bb96dc69b6bcf7bca9 P 571b1f82c62640baa5f92127434ff04e: Bootstrap starting.
I20260929 18:32:40.512578 5078 tablet_bootstrap.cc:654] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4eb628a7dcc2441aa375002c139dcace: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:40.513047 5076 tablet_bootstrap.cc:654] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4ee346edbe3a4443bb23cccccd8b9f75: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:40.513769 5077 tablet_bootstrap.cc:654] T fb01ea03b01140bb96dc69b6bcf7bca9 P 571b1f82c62640baa5f92127434ff04e: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:40.514246 5078 tablet_bootstrap.cc:492] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4eb628a7dcc2441aa375002c139dcace: No bootstrap required, opened a new log
I20260929 18:32:40.514353 5078 ts_tablet_manager.cc:1397] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4eb628a7dcc2441aa375002c139dcace: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260929 18:32:40.514386 5076 tablet_bootstrap.cc:492] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4ee346edbe3a4443bb23cccccd8b9f75: No bootstrap required, opened a new log
I20260929 18:32:40.514479 5076 ts_tablet_manager.cc:1397] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4ee346edbe3a4443bb23cccccd8b9f75: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:32:40.514940 5076 raft_consensus.cc:348] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:40.514940 5078 raft_consensus.cc:348] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4eb628a7dcc2441aa375002c139dcace [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:40.515093 5078 raft_consensus.cc:374] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4eb628a7dcc2441aa375002c139dcace [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:40.515093 5076 raft_consensus.cc:374] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:40.515146 5076 raft_consensus.cc:729] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ee346edbe3a4443bb23cccccd8b9f75, State: Initialized, Role: FOLLOWER
I20260929 18:32:40.515146 5078 raft_consensus.cc:729] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4eb628a7dcc2441aa375002c139dcace [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb628a7dcc2441aa375002c139dcace, State: Initialized, Role: FOLLOWER
I20260929 18:32:40.515307 5078 consensus_queue.cc:260] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4eb628a7dcc2441aa375002c139dcace [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:40.515328 5076 consensus_queue.cc:260] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4ee346edbe3a4443bb23cccccd8b9f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:40.515452 5077 tablet_bootstrap.cc:492] T fb01ea03b01140bb96dc69b6bcf7bca9 P 571b1f82c62640baa5f92127434ff04e: No bootstrap required, opened a new log
I20260929 18:32:40.515544 5077 ts_tablet_manager.cc:1397] T fb01ea03b01140bb96dc69b6bcf7bca9 P 571b1f82c62640baa5f92127434ff04e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:32:40.515700 5078 ts_tablet_manager.cc:1428] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4eb628a7dcc2441aa375002c139dcace: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 18:32:40.515622 5086 tablet_replica.cc:439] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4ee346edbe3a4443bb23cccccd8b9f75: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } } }
I20260929 18:32:40.515776 5086 tablet_replica.cc:442] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4ee346edbe3a4443bb23cccccd8b9f75: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:32:40.515924 5084 tablet_replica.cc:439] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4eb628a7dcc2441aa375002c139dcace: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } } }
I20260929 18:32:40.516042 5076 ts_tablet_manager.cc:1428] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4ee346edbe3a4443bb23cccccd8b9f75: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:32:40.516099 5084 tablet_replica.cc:442] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4eb628a7dcc2441aa375002c139dcace: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:32:40.516286 5077 raft_consensus.cc:348] T fb01ea03b01140bb96dc69b6bcf7bca9 P 571b1f82c62640baa5f92127434ff04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:40.516456 5077 raft_consensus.cc:374] T fb01ea03b01140bb96dc69b6bcf7bca9 P 571b1f82c62640baa5f92127434ff04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:40.516496 5077 raft_consensus.cc:729] T fb01ea03b01140bb96dc69b6bcf7bca9 P 571b1f82c62640baa5f92127434ff04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 571b1f82c62640baa5f92127434ff04e, State: Initialized, Role: FOLLOWER
I20260929 18:32:40.516644 5077 consensus_queue.cc:260] T fb01ea03b01140bb96dc69b6bcf7bca9 P 571b1f82c62640baa5f92127434ff04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:40.516916 5083 tablet_replica.cc:439] T fb01ea03b01140bb96dc69b6bcf7bca9 P 571b1f82c62640baa5f92127434ff04e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } } }
I20260929 18:32:40.517060 5083 tablet_replica.cc:442] T fb01ea03b01140bb96dc69b6bcf7bca9 P 571b1f82c62640baa5f92127434ff04e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:32:40.517778 5077 ts_tablet_manager.cc:1428] T fb01ea03b01140bb96dc69b6bcf7bca9 P 571b1f82c62640baa5f92127434ff04e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 18:32:40.543777 5086 raft_consensus.cc:482] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:32:40.543955 5086 raft_consensus.cc:504] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:40.544508 5086 leader_election.cc:288] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4ee346edbe3a4443bb23cccccd8b9f75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4eb628a7dcc2441aa375002c139dcace (127.28.207.65:38705), 571b1f82c62640baa5f92127434ff04e (127.28.207.67:40275)
I20260929 18:32:40.544940 5002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb01ea03b01140bb96dc69b6bcf7bca9" candidate_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "571b1f82c62640baa5f92127434ff04e" is_pre_election: true
I20260929 18:32:40.545115 5002 raft_consensus.cc:2454] T fb01ea03b01140bb96dc69b6bcf7bca9 P 571b1f82c62640baa5f92127434ff04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ee346edbe3a4443bb23cccccd8b9f75 in term 0.
I20260929 18:32:40.545085 4722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb01ea03b01140bb96dc69b6bcf7bca9" candidate_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb628a7dcc2441aa375002c139dcace" is_pre_election: true
I20260929 18:32:40.545333 4722 raft_consensus.cc:2454] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4eb628a7dcc2441aa375002c139dcace [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ee346edbe3a4443bb23cccccd8b9f75 in term 0.
I20260929 18:32:40.545419 4796 leader_election.cc:302] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4ee346edbe3a4443bb23cccccd8b9f75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ee346edbe3a4443bb23cccccd8b9f75, 571b1f82c62640baa5f92127434ff04e; no voters:
I20260929 18:32:40.545619 5086 raft_consensus.cc:2781] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:32:40.545679 5086 raft_consensus.cc:482] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:32:40.545724 5086 raft_consensus.cc:3037] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:40.546918 5086 raft_consensus.cc:504] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:40.547305 5086 leader_election.cc:288] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4ee346edbe3a4443bb23cccccd8b9f75 [CANDIDATE]: Term 1 election: Requested vote from peers 4eb628a7dcc2441aa375002c139dcace (127.28.207.65:38705), 571b1f82c62640baa5f92127434ff04e (127.28.207.67:40275)
I20260929 18:32:40.547705 4722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb01ea03b01140bb96dc69b6bcf7bca9" candidate_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb628a7dcc2441aa375002c139dcace"
I20260929 18:32:40.547778 5002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb01ea03b01140bb96dc69b6bcf7bca9" candidate_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "571b1f82c62640baa5f92127434ff04e"
I20260929 18:32:40.547878 5002 raft_consensus.cc:3037] T fb01ea03b01140bb96dc69b6bcf7bca9 P 571b1f82c62640baa5f92127434ff04e [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:40.547878 4722 raft_consensus.cc:3037] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4eb628a7dcc2441aa375002c139dcace [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:40.548990 4722 raft_consensus.cc:2454] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4eb628a7dcc2441aa375002c139dcace [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ee346edbe3a4443bb23cccccd8b9f75 in term 1.
I20260929 18:32:40.548990 5002 raft_consensus.cc:2454] T fb01ea03b01140bb96dc69b6bcf7bca9 P 571b1f82c62640baa5f92127434ff04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ee346edbe3a4443bb23cccccd8b9f75 in term 1.
I20260929 18:32:40.549345 4793 leader_election.cc:302] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4ee346edbe3a4443bb23cccccd8b9f75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb628a7dcc2441aa375002c139dcace, 4ee346edbe3a4443bb23cccccd8b9f75; no voters:
I20260929 18:32:40.549592 5086 raft_consensus.cc:2781] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:32:40.549695 5086 raft_consensus.cc:686] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 1 LEADER]: Becoming Leader. State: Replica: 4ee346edbe3a4443bb23cccccd8b9f75, State: Running, Role: LEADER
I20260929 18:32:40.550082 5086 consensus_queue.cc:237] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4ee346edbe3a4443bb23cccccd8b9f75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:40.551321 5082 tablet_replica.cc:439] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4ee346edbe3a4443bb23cccccd8b9f75: TxnStatusTablet state changed. Reason: New leader 4ee346edbe3a4443bb23cccccd8b9f75. Latest consensus state: current_term: 1 leader_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } } }
I20260929 18:32:40.551507 5082 tablet_replica.cc:442] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4ee346edbe3a4443bb23cccccd8b9f75: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:32:40.551996 5116 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 18:32:40.552201 5116 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 18:32:40.552568 4561 catalog_manager.cc:5495] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4ee346edbe3a4443bb23cccccd8b9f75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ee346edbe3a4443bb23cccccd8b9f75 (127.28.207.66). New cstate: current_term: 1 leader_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } health_report { overall_health: UNKNOWN } } }
I20260929 18:32:40.577940 5111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:40.573190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:38545 (local address 127.0.0.1:33164)
0929 18:32:40.573358 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:40.573377 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:40.573510 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:40.573919 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:40.573922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:40.573930 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:32:40.574075 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:40.574081 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.575265 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.575268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:40.576180 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:40.576191 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.576867 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.576871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:40.576961 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:40.577029 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:32:40.577577 (+ 548us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:32:40.577596 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 18:32:40.577679 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 18:32:40.577789 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 18:32:40.577942 5073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:40.573293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33164 (local address 127.28.207.66:38545)
0929 18:32:40.573475 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:40.573480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:40.573500 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:32:40.573663 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:40.573666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:40.573728 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:40.573838 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:32:40.574228 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.575084 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.576333 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.576714 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.576833 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:40.577350 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:32:40.577444 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:32:40.577536 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:40.577713 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 18:32:40.577778 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 18:32:40.581693 4722 raft_consensus.cc:1261] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4eb628a7dcc2441aa375002c139dcace [term 1 FOLLOWER]: Refusing update from remote peer 4ee346edbe3a4443bb23cccccd8b9f75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:32:40.581713 5002 raft_consensus.cc:1261] T fb01ea03b01140bb96dc69b6bcf7bca9 P 571b1f82c62640baa5f92127434ff04e [term 1 FOLLOWER]: Refusing update from remote peer 4ee346edbe3a4443bb23cccccd8b9f75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:32:40.582288 5082 consensus_queue.cc:1035] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4ee346edbe3a4443bb23cccccd8b9f75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:32:40.582607 5086 consensus_queue.cc:1035] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4ee346edbe3a4443bb23cccccd8b9f75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:32:40.583917 5084 tablet_replica.cc:439] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4eb628a7dcc2441aa375002c139dcace: TxnStatusTablet state changed. Reason: New leader 4ee346edbe3a4443bb23cccccd8b9f75. Latest consensus state: current_term: 1 leader_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } } }
I20260929 18:32:40.584057 5084 tablet_replica.cc:442] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4eb628a7dcc2441aa375002c139dcace: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:32:40.584137 5083 tablet_replica.cc:439] T fb01ea03b01140bb96dc69b6bcf7bca9 P 571b1f82c62640baa5f92127434ff04e: TxnStatusTablet state changed. Reason: New leader 4ee346edbe3a4443bb23cccccd8b9f75. Latest consensus state: current_term: 1 leader_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } } }
I20260929 18:32:40.584260 5083 tablet_replica.cc:442] T fb01ea03b01140bb96dc69b6bcf7bca9 P 571b1f82c62640baa5f92127434ff04e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:32:40.586696 5086 tablet_replica.cc:439] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4ee346edbe3a4443bb23cccccd8b9f75: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } } }
I20260929 18:32:40.586823 5086 tablet_replica.cc:442] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4ee346edbe3a4443bb23cccccd8b9f75: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:32:40.587651 5090 mvcc.cc:205] Tried to move back new op lower bound from 7334734891338280960 to 7334734891215478784. Current Snapshot: MvccSnapshot[applied={T|T < 7334734891338280960}]
I20260929 18:32:40.587839 5083 tablet_replica.cc:439] T fb01ea03b01140bb96dc69b6bcf7bca9 P 571b1f82c62640baa5f92127434ff04e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } } }
I20260929 18:32:40.587932 5083 tablet_replica.cc:442] T fb01ea03b01140bb96dc69b6bcf7bca9 P 571b1f82c62640baa5f92127434ff04e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:32:40.592001 5082 tablet_replica.cc:439] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4ee346edbe3a4443bb23cccccd8b9f75: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } } }
I20260929 18:32:40.592164 5082 tablet_replica.cc:442] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4ee346edbe3a4443bb23cccccd8b9f75: This TxnStatusTablet replica's current role is: LEADER
I20260929 18:32:40.593269 5084 tablet_replica.cc:439] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4eb628a7dcc2441aa375002c139dcace: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } } }
I20260929 18:32:40.593420 5084 tablet_replica.cc:442] T fb01ea03b01140bb96dc69b6bcf7bca9 P 4eb628a7dcc2441aa375002c139dcace: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 18:32:40.606693 5122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:40.601626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:38545 (local address 127.28.207.66:45653)
0929 18:32:40.602056 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:40.602071 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:40.602218 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:40.602523 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:40.602526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:40.602534 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:32:40.602678 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:40.602683 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.603727 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.603730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:40.604777 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:40.604788 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.605420 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.605423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:40.605705 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:40.605839 (+ 134us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:32:40.606312 (+ 473us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:32:40.606319 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 18:32:40.606411 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 18:32:40.606498 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":3,"thread_start_us":173,"threads_started":1}
I20260929 18:32:40.606698 5073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:40.601701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:45653 (local address 127.28.207.66:38545)
0929 18:32:40.601880 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:40.601885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:40.602169 (+ 284us) server_negotiation.cc:407] Connection header received
0929 18:32:40.602329 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:40.602332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:40.602386 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:40.602482 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:32:40.602784 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.603592 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.604914 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.605269 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.605387 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:40.606105 (+ 718us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:32:40.606200 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:32:40.606281 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:40.606446 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 18:32:40.606504 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":49}
I20260929 18:32:40.615504 5002 raft_consensus.cc:1261] T f173539a9a314366938e389af35dcd80 P 571b1f82c62640baa5f92127434ff04e [term 1 FOLLOWER]: Refusing update from remote peer 4ee346edbe3a4443bb23cccccd8b9f75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:32:40.615547 4722 raft_consensus.cc:1261] T f173539a9a314366938e389af35dcd80 P 4eb628a7dcc2441aa375002c139dcace [term 1 FOLLOWER]: Refusing update from remote peer 4ee346edbe3a4443bb23cccccd8b9f75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:32:40.615988 5082 consensus_queue.cc:1035] T f173539a9a314366938e389af35dcd80 P 4ee346edbe3a4443bb23cccccd8b9f75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:32:40.616205 5086 consensus_queue.cc:1035] T f173539a9a314366938e389af35dcd80 P 4ee346edbe3a4443bb23cccccd8b9f75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:32:40.623487 5089 mvcc.cc:205] Tried to move back new op lower bound from 7334734891476992000 to 7334734890753916928. Current Snapshot: MvccSnapshot[applied={T|T < 7334734891476992000}]
I20260929 18:32:40.670828 4561 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:32:40.671136 4561 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:32:40.673741 4561 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf94f8dce239472093ae97936b750185: Sending DeleteTablet for 3 replicas of tablet f173539a9a314366938e389af35dcd80
I20260929 18:32:40.674940 4982 tablet_service.cc:1514] Processing DeleteTablet for tablet f173539a9a314366938e389af35dcd80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:40 UTC) from {username='oryx', principal='oryx/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:33892
I20260929 18:32:40.674953 4702 tablet_service.cc:1514] Processing DeleteTablet for tablet f173539a9a314366938e389af35dcd80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:40 UTC) from {username='oryx', principal='oryx/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:40364
I20260929 18:32:40.675398 5099 tablet_replica.cc:330] T f173539a9a314366938e389af35dcd80 P 4eb628a7dcc2441aa375002c139dcace: stopping tablet replica
I20260929 18:32:40.675613 5099 raft_consensus.cc:2229] T f173539a9a314366938e389af35dcd80 P 4eb628a7dcc2441aa375002c139dcace [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:32:40.675807 5099 raft_consensus.cc:2258] T f173539a9a314366938e389af35dcd80 P 4eb628a7dcc2441aa375002c139dcace [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:40.676043 5100 tablet_replica.cc:330] T f173539a9a314366938e389af35dcd80 P 571b1f82c62640baa5f92127434ff04e: stopping tablet replica
I20260929 18:32:40.676198 5100 raft_consensus.cc:2229] T f173539a9a314366938e389af35dcd80 P 571b1f82c62640baa5f92127434ff04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:32:40.676371 5100 raft_consensus.cc:2258] T f173539a9a314366938e389af35dcd80 P 571b1f82c62640baa5f92127434ff04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:40.677487 5100 ts_tablet_manager.cc:1905] T f173539a9a314366938e389af35dcd80 P 571b1f82c62640baa5f92127434ff04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:32:40.677891 5099 ts_tablet_manager.cc:1905] T f173539a9a314366938e389af35dcd80 P 4eb628a7dcc2441aa375002c139dcace: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:32:40.681767 5099 ts_tablet_manager.cc:1918] T f173539a9a314366938e389af35dcd80 P 4eb628a7dcc2441aa375002c139dcace: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:32:40.681896 5099 log.cc:1196] T f173539a9a314366938e389af35dcd80 P 4eb628a7dcc2441aa375002c139dcace: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-0/wal/wals/f173539a9a314366938e389af35dcd80
I20260929 18:32:40.682202 5099 ts_tablet_manager.cc:1939] T f173539a9a314366938e389af35dcd80 P 4eb628a7dcc2441aa375002c139dcace: Deleting consensus metadata
I20260929 18:32:40.683126 4842 tablet_service.cc:1514] Processing DeleteTablet for tablet f173539a9a314366938e389af35dcd80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:40 UTC) from {username='oryx', principal='oryx/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:33144
I20260929 18:32:40.683565 5101 tablet_replica.cc:330] T f173539a9a314366938e389af35dcd80 P 4ee346edbe3a4443bb23cccccd8b9f75: stopping tablet replica
I20260929 18:32:40.683657 4549 catalog_manager.cc:4841] TS 4eb628a7dcc2441aa375002c139dcace (127.28.207.65:38705): tablet f173539a9a314366938e389af35dcd80 (table test-table [id=d95aa78406654e94a91cbc942c0ed3f7]) successfully deleted
I20260929 18:32:40.683776 5101 raft_consensus.cc:2229] T f173539a9a314366938e389af35dcd80 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:32:40.684087 5101 raft_consensus.cc:2258] T f173539a9a314366938e389af35dcd80 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:40.684798 5100 ts_tablet_manager.cc:1918] T f173539a9a314366938e389af35dcd80 P 571b1f82c62640baa5f92127434ff04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:32:40.684914 5100 log.cc:1196] T f173539a9a314366938e389af35dcd80 P 571b1f82c62640baa5f92127434ff04e: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-2/wal/wals/f173539a9a314366938e389af35dcd80
I20260929 18:32:40.685182 5100 ts_tablet_manager.cc:1939] T f173539a9a314366938e389af35dcd80 P 571b1f82c62640baa5f92127434ff04e: Deleting consensus metadata
I20260929 18:32:40.685228 5101 ts_tablet_manager.cc:1905] T f173539a9a314366938e389af35dcd80 P 4ee346edbe3a4443bb23cccccd8b9f75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:32:40.686228 4552 catalog_manager.cc:4841] TS 571b1f82c62640baa5f92127434ff04e (127.28.207.67:40275): tablet f173539a9a314366938e389af35dcd80 (table test-table [id=d95aa78406654e94a91cbc942c0ed3f7]) successfully deleted
I20260929 18:32:40.689910 5101 ts_tablet_manager.cc:1918] T f173539a9a314366938e389af35dcd80 P 4ee346edbe3a4443bb23cccccd8b9f75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:32:40.690006 5101 log.cc:1196] T f173539a9a314366938e389af35dcd80 P 4ee346edbe3a4443bb23cccccd8b9f75: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-1/wal/wals/f173539a9a314366938e389af35dcd80
I20260929 18:32:40.690218 5101 ts_tablet_manager.cc:1939] T f173539a9a314366938e389af35dcd80 P 4ee346edbe3a4443bb23cccccd8b9f75: Deleting consensus metadata
I20260929 18:32:40.691457 4550 catalog_manager.cc:4841] TS 4ee346edbe3a4443bb23cccccd8b9f75 (127.28.207.66:38545): tablet f173539a9a314366938e389af35dcd80 (table test-table [id=d95aa78406654e94a91cbc942c0ed3f7]) successfully deleted
Sep 29 18:32:40 dist-test-slave-qcpv krb5kdc[4520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 18:32:40 dist-test-slave-qcpv krb5kdc[4520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:40 dist-test-slave-qcpv krb5kdc[4520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 18:32:40.714912 4614 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:40.706415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32988 (local address 127.28.207.126:37575)
0929 18:32:40.706603 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:40.706609 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:40.707082 (+ 473us) server_negotiation.cc:407] Connection header received
0929 18:32:40.707328 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:40.707331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:40.707403 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:40.707504 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:40.708469 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.709291 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.710488 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.710740 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.710915 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:40.714749 (+ 3834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32988: BlockingRecv error: recv got EOF from 127.0.0.1:32988 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 29 18:32:40 dist-test-slave-qcpv krb5kdc[4520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM
I20260929 18:32:40.729416 4614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:40.716646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33004 (local address 127.28.207.126:37575)
0929 18:32:40.716793 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:40.716796 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:40.716958 (+ 162us) server_negotiation.cc:407] Connection header received
0929 18:32:40.717046 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:40.717049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:40.717102 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:40.717181 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:40.717965 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.718611 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.719860 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.720121 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.720232 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:40.723779 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:40.723822 (+ 43us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:40.723826 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:40.723867 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:40.726780 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:40.727777 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:40.727781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:40.727783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:40.727889 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:40.728315 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:40.728318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:40.728319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:40.728533 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:40.728650 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:40.729117 (+ 467us) server_negotiation.cc:299] Negotiation successful
0929 18:32:40.729252 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 18:32:40.732911 4561 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33004:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:32:40.733392 4561 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:32:40.739284 4842 tablet_service.cc:1467] Processing CreateTablet for tablet e5ec4b16857a4f5fa171e06631bf2a55 (DEFAULT_TABLE table=test-table [id=6c10e99dbca6468bab51f82a1d359c15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:40.739311 4702 tablet_service.cc:1467] Processing CreateTablet for tablet e5ec4b16857a4f5fa171e06631bf2a55 (DEFAULT_TABLE table=test-table [id=6c10e99dbca6468bab51f82a1d359c15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:40.739465 4982 tablet_service.cc:1467] Processing CreateTablet for tablet e5ec4b16857a4f5fa171e06631bf2a55 (DEFAULT_TABLE table=test-table [id=6c10e99dbca6468bab51f82a1d359c15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:40.739741 4702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ec4b16857a4f5fa171e06631bf2a55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:40.739863 4842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ec4b16857a4f5fa171e06631bf2a55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:40.739842 4982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5ec4b16857a4f5fa171e06631bf2a55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:40.742007 5078 tablet_bootstrap.cc:492] T e5ec4b16857a4f5fa171e06631bf2a55 P 4eb628a7dcc2441aa375002c139dcace: Bootstrap starting.
I20260929 18:32:40.742555 5077 tablet_bootstrap.cc:492] T e5ec4b16857a4f5fa171e06631bf2a55 P 571b1f82c62640baa5f92127434ff04e: Bootstrap starting.
I20260929 18:32:40.743140 5078 tablet_bootstrap.cc:654] T e5ec4b16857a4f5fa171e06631bf2a55 P 4eb628a7dcc2441aa375002c139dcace: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:40.743233 5076 tablet_bootstrap.cc:492] T e5ec4b16857a4f5fa171e06631bf2a55 P 4ee346edbe3a4443bb23cccccd8b9f75: Bootstrap starting.
I20260929 18:32:40.743628 5077 tablet_bootstrap.cc:654] T e5ec4b16857a4f5fa171e06631bf2a55 P 571b1f82c62640baa5f92127434ff04e: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:40.744220 5076 tablet_bootstrap.cc:654] T e5ec4b16857a4f5fa171e06631bf2a55 P 4ee346edbe3a4443bb23cccccd8b9f75: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:40.744745 5078 tablet_bootstrap.cc:492] T e5ec4b16857a4f5fa171e06631bf2a55 P 4eb628a7dcc2441aa375002c139dcace: No bootstrap required, opened a new log
I20260929 18:32:40.744867 5078 ts_tablet_manager.cc:1397] T e5ec4b16857a4f5fa171e06631bf2a55 P 4eb628a7dcc2441aa375002c139dcace: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:32:40.745059 5077 tablet_bootstrap.cc:492] T e5ec4b16857a4f5fa171e06631bf2a55 P 571b1f82c62640baa5f92127434ff04e: No bootstrap required, opened a new log
I20260929 18:32:40.745149 5077 ts_tablet_manager.cc:1397] T e5ec4b16857a4f5fa171e06631bf2a55 P 571b1f82c62640baa5f92127434ff04e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:32:40.745471 5078 raft_consensus.cc:348] T e5ec4b16857a4f5fa171e06631bf2a55 P 4eb628a7dcc2441aa375002c139dcace [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:40.745626 5078 raft_consensus.cc:374] T e5ec4b16857a4f5fa171e06631bf2a55 P 4eb628a7dcc2441aa375002c139dcace [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:40.745661 5078 raft_consensus.cc:729] T e5ec4b16857a4f5fa171e06631bf2a55 P 4eb628a7dcc2441aa375002c139dcace [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb628a7dcc2441aa375002c139dcace, State: Initialized, Role: FOLLOWER
I20260929 18:32:40.745667 5076 tablet_bootstrap.cc:492] T e5ec4b16857a4f5fa171e06631bf2a55 P 4ee346edbe3a4443bb23cccccd8b9f75: No bootstrap required, opened a new log
I20260929 18:32:40.745752 5076 ts_tablet_manager.cc:1397] T e5ec4b16857a4f5fa171e06631bf2a55 P 4ee346edbe3a4443bb23cccccd8b9f75: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 18:32:40.745831 5078 consensus_queue.cc:260] T e5ec4b16857a4f5fa171e06631bf2a55 P 4eb628a7dcc2441aa375002c139dcace [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:40.745936 5077 raft_consensus.cc:348] T e5ec4b16857a4f5fa171e06631bf2a55 P 571b1f82c62640baa5f92127434ff04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:40.746054 5077 raft_consensus.cc:374] T e5ec4b16857a4f5fa171e06631bf2a55 P 571b1f82c62640baa5f92127434ff04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:40.746086 5077 raft_consensus.cc:729] T e5ec4b16857a4f5fa171e06631bf2a55 P 571b1f82c62640baa5f92127434ff04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 571b1f82c62640baa5f92127434ff04e, State: Initialized, Role: FOLLOWER
I20260929 18:32:40.746204 5077 consensus_queue.cc:260] T e5ec4b16857a4f5fa171e06631bf2a55 P 571b1f82c62640baa5f92127434ff04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:40.746253 5078 ts_tablet_manager.cc:1428] T e5ec4b16857a4f5fa171e06631bf2a55 P 4eb628a7dcc2441aa375002c139dcace: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:32:40.746263 5076 raft_consensus.cc:348] T e5ec4b16857a4f5fa171e06631bf2a55 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:40.746407 5076 raft_consensus.cc:374] T e5ec4b16857a4f5fa171e06631bf2a55 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:40.746440 5077 ts_tablet_manager.cc:1428] T e5ec4b16857a4f5fa171e06631bf2a55 P 571b1f82c62640baa5f92127434ff04e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 18:32:40.746450 5076 raft_consensus.cc:729] T e5ec4b16857a4f5fa171e06631bf2a55 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ee346edbe3a4443bb23cccccd8b9f75, State: Initialized, Role: FOLLOWER
I20260929 18:32:40.746598 5076 consensus_queue.cc:260] T e5ec4b16857a4f5fa171e06631bf2a55 P 4ee346edbe3a4443bb23cccccd8b9f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:40.746929 5076 ts_tablet_manager.cc:1428] T e5ec4b16857a4f5fa171e06631bf2a55 P 4ee346edbe3a4443bb23cccccd8b9f75: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 18:32:40.822273 5082 raft_consensus.cc:482] T e5ec4b16857a4f5fa171e06631bf2a55 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:32:40.822494 5082 raft_consensus.cc:504] T e5ec4b16857a4f5fa171e06631bf2a55 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:40.823127 5082 leader_election.cc:288] T e5ec4b16857a4f5fa171e06631bf2a55 P 4ee346edbe3a4443bb23cccccd8b9f75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4eb628a7dcc2441aa375002c139dcace (127.28.207.65:38705), 571b1f82c62640baa5f92127434ff04e (127.28.207.67:40275)
I20260929 18:32:40.823822 5002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ec4b16857a4f5fa171e06631bf2a55" candidate_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "571b1f82c62640baa5f92127434ff04e" is_pre_election: true
I20260929 18:32:40.823820 4722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ec4b16857a4f5fa171e06631bf2a55" candidate_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb628a7dcc2441aa375002c139dcace" is_pre_election: true
I20260929 18:32:40.824019 5002 raft_consensus.cc:2454] T e5ec4b16857a4f5fa171e06631bf2a55 P 571b1f82c62640baa5f92127434ff04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ee346edbe3a4443bb23cccccd8b9f75 in term 0.
I20260929 18:32:40.824023 4722 raft_consensus.cc:2454] T e5ec4b16857a4f5fa171e06631bf2a55 P 4eb628a7dcc2441aa375002c139dcace [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ee346edbe3a4443bb23cccccd8b9f75 in term 0.
I20260929 18:32:40.824451 4793 leader_election.cc:302] T e5ec4b16857a4f5fa171e06631bf2a55 P 4ee346edbe3a4443bb23cccccd8b9f75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb628a7dcc2441aa375002c139dcace, 4ee346edbe3a4443bb23cccccd8b9f75; no voters:
I20260929 18:32:40.824669 5082 raft_consensus.cc:2781] T e5ec4b16857a4f5fa171e06631bf2a55 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:32:40.824757 5082 raft_consensus.cc:482] T e5ec4b16857a4f5fa171e06631bf2a55 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:32:40.824817 5082 raft_consensus.cc:3037] T e5ec4b16857a4f5fa171e06631bf2a55 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:40.826035 5082 raft_consensus.cc:504] T e5ec4b16857a4f5fa171e06631bf2a55 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:40.826443 5082 leader_election.cc:288] T e5ec4b16857a4f5fa171e06631bf2a55 P 4ee346edbe3a4443bb23cccccd8b9f75 [CANDIDATE]: Term 1 election: Requested vote from peers 4eb628a7dcc2441aa375002c139dcace (127.28.207.65:38705), 571b1f82c62640baa5f92127434ff04e (127.28.207.67:40275)
I20260929 18:32:40.826997 4722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ec4b16857a4f5fa171e06631bf2a55" candidate_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb628a7dcc2441aa375002c139dcace"
I20260929 18:32:40.827037 5002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5ec4b16857a4f5fa171e06631bf2a55" candidate_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "571b1f82c62640baa5f92127434ff04e"
I20260929 18:32:40.827169 4722 raft_consensus.cc:3037] T e5ec4b16857a4f5fa171e06631bf2a55 P 4eb628a7dcc2441aa375002c139dcace [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:40.827172 5002 raft_consensus.cc:3037] T e5ec4b16857a4f5fa171e06631bf2a55 P 571b1f82c62640baa5f92127434ff04e [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:40.828510 4722 raft_consensus.cc:2454] T e5ec4b16857a4f5fa171e06631bf2a55 P 4eb628a7dcc2441aa375002c139dcace [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ee346edbe3a4443bb23cccccd8b9f75 in term 1.
I20260929 18:32:40.828871 5002 raft_consensus.cc:2454] T e5ec4b16857a4f5fa171e06631bf2a55 P 571b1f82c62640baa5f92127434ff04e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ee346edbe3a4443bb23cccccd8b9f75 in term 1.
I20260929 18:32:40.829037 4793 leader_election.cc:302] T e5ec4b16857a4f5fa171e06631bf2a55 P 4ee346edbe3a4443bb23cccccd8b9f75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb628a7dcc2441aa375002c139dcace, 4ee346edbe3a4443bb23cccccd8b9f75; no voters:
I20260929 18:32:40.829388 5082 raft_consensus.cc:2781] T e5ec4b16857a4f5fa171e06631bf2a55 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:32:40.829514 5082 raft_consensus.cc:686] T e5ec4b16857a4f5fa171e06631bf2a55 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 1 LEADER]: Becoming Leader. State: Replica: 4ee346edbe3a4443bb23cccccd8b9f75, State: Running, Role: LEADER
I20260929 18:32:40.829698 5082 consensus_queue.cc:237] T e5ec4b16857a4f5fa171e06631bf2a55 P 4ee346edbe3a4443bb23cccccd8b9f75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:40.831665 4564 catalog_manager.cc:5495] T e5ec4b16857a4f5fa171e06631bf2a55 P 4ee346edbe3a4443bb23cccccd8b9f75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ee346edbe3a4443bb23cccccd8b9f75 (127.28.207.66). New cstate: current_term: 1 leader_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } health_report { overall_health: UNKNOWN } } }
I20260929 18:32:40.858327 5073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:40.853228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33176 (local address 127.28.207.66:38545)
0929 18:32:40.853413 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:40.853418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:40.853493 (+ 75us) server_negotiation.cc:407] Connection header received
0929 18:32:40.853864 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:40.853867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:40.853944 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:40.854077 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:32:40.854612 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.855534 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.857014 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:40.857404 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:40.857519 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:40.857848 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:32:40.857936 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:32:40.858012 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:40.858110 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 18:32:40.858165 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260929 18:32:40.861027 4722 raft_consensus.cc:1261] T e5ec4b16857a4f5fa171e06631bf2a55 P 4eb628a7dcc2441aa375002c139dcace [term 1 FOLLOWER]: Refusing update from remote peer 4ee346edbe3a4443bb23cccccd8b9f75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:32:40.861027 5002 raft_consensus.cc:1261] T e5ec4b16857a4f5fa171e06631bf2a55 P 571b1f82c62640baa5f92127434ff04e [term 1 FOLLOWER]: Refusing update from remote peer 4ee346edbe3a4443bb23cccccd8b9f75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:32:40.861649 5082 consensus_queue.cc:1035] T e5ec4b16857a4f5fa171e06631bf2a55 P 4ee346edbe3a4443bb23cccccd8b9f75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:32:40.861945 5086 consensus_queue.cc:1035] T e5ec4b16857a4f5fa171e06631bf2a55 P 4ee346edbe3a4443bb23cccccd8b9f75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:32:40.871704 4565 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:32:40.871896 4565 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:32:40.874202 4565 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf94f8dce239472093ae97936b750185: Sending DeleteTablet for 3 replicas of tablet e5ec4b16857a4f5fa171e06631bf2a55
I20260929 18:32:40.874957 4842 tablet_service.cc:1514] Processing DeleteTablet for tablet e5ec4b16857a4f5fa171e06631bf2a55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:40 UTC) from {username='oryx', principal='oryx/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:33144
I20260929 18:32:40.874995 4982 tablet_service.cc:1514] Processing DeleteTablet for tablet e5ec4b16857a4f5fa171e06631bf2a55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:40 UTC) from {username='oryx', principal='oryx/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:33892
I20260929 18:32:40.875193 5100 tablet_replica.cc:330] T e5ec4b16857a4f5fa171e06631bf2a55 P 571b1f82c62640baa5f92127434ff04e: stopping tablet replica
I20260929 18:32:40.875185 4702 tablet_service.cc:1514] Processing DeleteTablet for tablet e5ec4b16857a4f5fa171e06631bf2a55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:40 UTC) from {username='oryx', principal='oryx/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:40364
I20260929 18:32:40.875314 5100 raft_consensus.cc:2229] T e5ec4b16857a4f5fa171e06631bf2a55 P 571b1f82c62640baa5f92127434ff04e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:32:40.875372 5099 tablet_replica.cc:330] T e5ec4b16857a4f5fa171e06631bf2a55 P 4eb628a7dcc2441aa375002c139dcace: stopping tablet replica
I20260929 18:32:40.875386 5101 tablet_replica.cc:330] T e5ec4b16857a4f5fa171e06631bf2a55 P 4ee346edbe3a4443bb23cccccd8b9f75: stopping tablet replica
I20260929 18:32:40.875456 5100 raft_consensus.cc:2258] T e5ec4b16857a4f5fa171e06631bf2a55 P 571b1f82c62640baa5f92127434ff04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:40.875514 5099 raft_consensus.cc:2229] T e5ec4b16857a4f5fa171e06631bf2a55 P 4eb628a7dcc2441aa375002c139dcace [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:32:40.875587 5101 raft_consensus.cc:2229] T e5ec4b16857a4f5fa171e06631bf2a55 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:32:40.875643 5099 raft_consensus.cc:2258] T e5ec4b16857a4f5fa171e06631bf2a55 P 4eb628a7dcc2441aa375002c139dcace [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:40.875824 5101 raft_consensus.cc:2258] T e5ec4b16857a4f5fa171e06631bf2a55 P 4ee346edbe3a4443bb23cccccd8b9f75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:40.876555 5099 ts_tablet_manager.cc:1905] T e5ec4b16857a4f5fa171e06631bf2a55 P 4eb628a7dcc2441aa375002c139dcace: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:32:40.876555 5100 ts_tablet_manager.cc:1905] T e5ec4b16857a4f5fa171e06631bf2a55 P 571b1f82c62640baa5f92127434ff04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:32:40.876581 5101 ts_tablet_manager.cc:1905] T e5ec4b16857a4f5fa171e06631bf2a55 P 4ee346edbe3a4443bb23cccccd8b9f75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:32:40.878834 5101 ts_tablet_manager.cc:1918] T e5ec4b16857a4f5fa171e06631bf2a55 P 4ee346edbe3a4443bb23cccccd8b9f75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:32:40.878996 5101 log.cc:1196] T e5ec4b16857a4f5fa171e06631bf2a55 P 4ee346edbe3a4443bb23cccccd8b9f75: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-1/wal/wals/e5ec4b16857a4f5fa171e06631bf2a55
I20260929 18:32:40.879271 5101 ts_tablet_manager.cc:1939] T e5ec4b16857a4f5fa171e06631bf2a55 P 4ee346edbe3a4443bb23cccccd8b9f75: Deleting consensus metadata
I20260929 18:32:40.879926 5099 ts_tablet_manager.cc:1918] T e5ec4b16857a4f5fa171e06631bf2a55 P 4eb628a7dcc2441aa375002c139dcace: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:32:40.879926 5100 ts_tablet_manager.cc:1918] T e5ec4b16857a4f5fa171e06631bf2a55 P 571b1f82c62640baa5f92127434ff04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:32:40.880048 5100 log.cc:1196] T e5ec4b16857a4f5fa171e06631bf2a55 P 571b1f82c62640baa5f92127434ff04e: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-2/wal/wals/e5ec4b16857a4f5fa171e06631bf2a55
I20260929 18:32:40.880048 5099 log.cc:1196] T e5ec4b16857a4f5fa171e06631bf2a55 P 4eb628a7dcc2441aa375002c139dcace: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-0/wal/wals/e5ec4b16857a4f5fa171e06631bf2a55
I20260929 18:32:40.880056 4550 catalog_manager.cc:4841] TS 4ee346edbe3a4443bb23cccccd8b9f75 (127.28.207.66:38545): tablet e5ec4b16857a4f5fa171e06631bf2a55 (table test-table [id=6c10e99dbca6468bab51f82a1d359c15]) successfully deleted
I20260929 18:32:40.880354 5099 ts_tablet_manager.cc:1939] T e5ec4b16857a4f5fa171e06631bf2a55 P 4eb628a7dcc2441aa375002c139dcace: Deleting consensus metadata
I20260929 18:32:40.880354 5100 ts_tablet_manager.cc:1939] T e5ec4b16857a4f5fa171e06631bf2a55 P 571b1f82c62640baa5f92127434ff04e: Deleting consensus metadata
I20260929 18:32:40.881356 4565 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33004:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 18:32:40.881525 4552 catalog_manager.cc:4841] TS 571b1f82c62640baa5f92127434ff04e (127.28.207.67:40275): tablet e5ec4b16857a4f5fa171e06631bf2a55 (table test-table [id=6c10e99dbca6468bab51f82a1d359c15]) successfully deleted
I20260929 18:32:40.881774 4549 catalog_manager.cc:4841] TS 4eb628a7dcc2441aa375002c139dcace (127.28.207.65:38705): tablet e5ec4b16857a4f5fa171e06631bf2a55 (table test-table [id=6c10e99dbca6468bab51f82a1d359c15]) successfully deleted
W20260929 18:32:40.882019 4565 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:32:40.888386 4702 tablet_service.cc:1467] Processing CreateTablet for tablet d2b7cd739f0d4034b3411ba37301a08c (DEFAULT_TABLE table=test-table [id=bd5a9abf629b4f6a8ea268e3219a98b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:40.888587 4982 tablet_service.cc:1467] Processing CreateTablet for tablet d2b7cd739f0d4034b3411ba37301a08c (DEFAULT_TABLE table=test-table [id=bd5a9abf629b4f6a8ea268e3219a98b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:40.888723 4842 tablet_service.cc:1467] Processing CreateTablet for tablet d2b7cd739f0d4034b3411ba37301a08c (DEFAULT_TABLE table=test-table [id=bd5a9abf629b4f6a8ea268e3219a98b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:32:40.888837 4702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2b7cd739f0d4034b3411ba37301a08c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:40.888901 4982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2b7cd739f0d4034b3411ba37301a08c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:40.889093 4842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2b7cd739f0d4034b3411ba37301a08c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:40.891163 5076 tablet_bootstrap.cc:492] T d2b7cd739f0d4034b3411ba37301a08c P 4ee346edbe3a4443bb23cccccd8b9f75: Bootstrap starting.
I20260929 18:32:40.891860 5077 tablet_bootstrap.cc:492] T d2b7cd739f0d4034b3411ba37301a08c P 571b1f82c62640baa5f92127434ff04e: Bootstrap starting.
I20260929 18:32:40.892038 5076 tablet_bootstrap.cc:654] T d2b7cd739f0d4034b3411ba37301a08c P 4ee346edbe3a4443bb23cccccd8b9f75: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:40.892942 5077 tablet_bootstrap.cc:654] T d2b7cd739f0d4034b3411ba37301a08c P 571b1f82c62640baa5f92127434ff04e: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:40.893702 5078 tablet_bootstrap.cc:492] T d2b7cd739f0d4034b3411ba37301a08c P 4eb628a7dcc2441aa375002c139dcace: Bootstrap starting.
I20260929 18:32:40.893910 5076 tablet_bootstrap.cc:492] T d2b7cd739f0d4034b3411ba37301a08c P 4ee346edbe3a4443bb23cccccd8b9f75: No bootstrap required, opened a new log
I20260929 18:32:40.894009 5076 ts_tablet_manager.cc:1397] T d2b7cd739f0d4034b3411ba37301a08c P 4ee346edbe3a4443bb23cccccd8b9f75: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 18:32:40.894472 5076 raft_consensus.cc:348] T d2b7cd739f0d4034b3411ba37301a08c P 4ee346edbe3a4443bb23cccccd8b9f75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:40.894606 5076 raft_consensus.cc:374] T d2b7cd739f0d4034b3411ba37301a08c P 4ee346edbe3a4443bb23cccccd8b9f75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:40.894640 5076 raft_consensus.cc:729] T d2b7cd739f0d4034b3411ba37301a08c P 4ee346edbe3a4443bb23cccccd8b9f75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ee346edbe3a4443bb23cccccd8b9f75, State: Initialized, Role: FOLLOWER
I20260929 18:32:40.894774 5076 consensus_queue.cc:260] T d2b7cd739f0d4034b3411ba37301a08c P 4ee346edbe3a4443bb23cccccd8b9f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:40.894960 5078 tablet_bootstrap.cc:654] T d2b7cd739f0d4034b3411ba37301a08c P 4eb628a7dcc2441aa375002c139dcace: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:40.895169 5076 ts_tablet_manager.cc:1428] T d2b7cd739f0d4034b3411ba37301a08c P 4ee346edbe3a4443bb23cccccd8b9f75: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 18:32:40.896196 5077 tablet_bootstrap.cc:492] T d2b7cd739f0d4034b3411ba37301a08c P 571b1f82c62640baa5f92127434ff04e: No bootstrap required, opened a new log
I20260929 18:32:40.896296 5077 ts_tablet_manager.cc:1397] T d2b7cd739f0d4034b3411ba37301a08c P 571b1f82c62640baa5f92127434ff04e: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260929 18:32:40.896610 5078 tablet_bootstrap.cc:492] T d2b7cd739f0d4034b3411ba37301a08c P 4eb628a7dcc2441aa375002c139dcace: No bootstrap required, opened a new log
I20260929 18:32:40.896750 5078 ts_tablet_manager.cc:1397] T d2b7cd739f0d4034b3411ba37301a08c P 4eb628a7dcc2441aa375002c139dcace: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260929 18:32:40.896818 5077 raft_consensus.cc:348] T d2b7cd739f0d4034b3411ba37301a08c P 571b1f82c62640baa5f92127434ff04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:40.896948 5077 raft_consensus.cc:374] T d2b7cd739f0d4034b3411ba37301a08c P 571b1f82c62640baa5f92127434ff04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:40.896983 5077 raft_consensus.cc:729] T d2b7cd739f0d4034b3411ba37301a08c P 571b1f82c62640baa5f92127434ff04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 571b1f82c62640baa5f92127434ff04e, State: Initialized, Role: FOLLOWER
I20260929 18:32:40.897156 5077 consensus_queue.cc:260] T d2b7cd739f0d4034b3411ba37301a08c P 571b1f82c62640baa5f92127434ff04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:40.897372 5078 raft_consensus.cc:348] T d2b7cd739f0d4034b3411ba37301a08c P 4eb628a7dcc2441aa375002c139dcace [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:40.897464 5077 ts_tablet_manager.cc:1428] T d2b7cd739f0d4034b3411ba37301a08c P 571b1f82c62640baa5f92127434ff04e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:32:40.897506 5078 raft_consensus.cc:374] T d2b7cd739f0d4034b3411ba37301a08c P 4eb628a7dcc2441aa375002c139dcace [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:40.897542 5078 raft_consensus.cc:729] T d2b7cd739f0d4034b3411ba37301a08c P 4eb628a7dcc2441aa375002c139dcace [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb628a7dcc2441aa375002c139dcace, State: Initialized, Role: FOLLOWER
I20260929 18:32:40.897675 5078 consensus_queue.cc:260] T d2b7cd739f0d4034b3411ba37301a08c P 4eb628a7dcc2441aa375002c139dcace [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:40.898093 5078 ts_tablet_manager.cc:1428] T d2b7cd739f0d4034b3411ba37301a08c P 4eb628a7dcc2441aa375002c139dcace: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260929 18:32:41.113481 5083 raft_consensus.cc:482] T d2b7cd739f0d4034b3411ba37301a08c P 571b1f82c62640baa5f92127434ff04e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:32:41.113658 5083 raft_consensus.cc:504] T d2b7cd739f0d4034b3411ba37301a08c P 571b1f82c62640baa5f92127434ff04e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:41.115042 5083 leader_election.cc:288] T d2b7cd739f0d4034b3411ba37301a08c P 571b1f82c62640baa5f92127434ff04e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4eb628a7dcc2441aa375002c139dcace (127.28.207.65:38705), 4ee346edbe3a4443bb23cccccd8b9f75 (127.28.207.66:38545)
I20260929 18:32:41.118759 5086 raft_consensus.cc:482] T d2b7cd739f0d4034b3411ba37301a08c P 4ee346edbe3a4443bb23cccccd8b9f75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:32:41.120289 5086 raft_consensus.cc:504] T d2b7cd739f0d4034b3411ba37301a08c P 4ee346edbe3a4443bb23cccccd8b9f75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:41.121053 5086 leader_election.cc:288] T d2b7cd739f0d4034b3411ba37301a08c P 4ee346edbe3a4443bb23cccccd8b9f75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4eb628a7dcc2441aa375002c139dcace (127.28.207.65:38705), 571b1f82c62640baa5f92127434ff04e (127.28.207.67:40275)
I20260929 18:32:41.121695 4722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2b7cd739f0d4034b3411ba37301a08c" candidate_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb628a7dcc2441aa375002c139dcace" is_pre_election: true
I20260929 18:32:41.121775 5002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2b7cd739f0d4034b3411ba37301a08c" candidate_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "571b1f82c62640baa5f92127434ff04e" is_pre_election: true
I20260929 18:32:41.121869 4722 raft_consensus.cc:2454] T d2b7cd739f0d4034b3411ba37301a08c P 4eb628a7dcc2441aa375002c139dcace [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ee346edbe3a4443bb23cccccd8b9f75 in term 0.
I20260929 18:32:41.122174 5073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:41.115565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:55351 (local address 127.28.207.66:38545)
0929 18:32:41.115692 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:41.115698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:41.116053 (+ 355us) server_negotiation.cc:407] Connection header received
0929 18:32:41.116128 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:41.116131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:41.116200 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:41.116286 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:41.116778 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:41.117772 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:41.119971 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:41.120635 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:41.120748 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:41.120847 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:41.121391 (+ 544us) server_negotiation.cc:299] Negotiation successful
0929 18:32:41.121483 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 18:32:41.122215 4793 leader_election.cc:302] T d2b7cd739f0d4034b3411ba37301a08c P 4ee346edbe3a4443bb23cccccd8b9f75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb628a7dcc2441aa375002c139dcace, 4ee346edbe3a4443bb23cccccd8b9f75; no voters:
I20260929 18:32:41.122462 5149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:41.115404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:55133 (local address 127.28.207.65:38705)
0929 18:32:41.115772 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:41.115778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:41.115805 (+ 27us) server_negotiation.cc:407] Connection header received
0929 18:32:41.116000 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:41.116005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:41.116136 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:41.116264 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:32:41.116778 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:41.117787 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:41.119971 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:41.120728 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:41.122046 (+ 1318us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:41.122112 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:41.122154 (+ 42us) server_negotiation.cc:299] Negotiation successful
0929 18:32:41.122250 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":114,"threads_started":1}
I20260929 18:32:41.121912 5002 raft_consensus.cc:2454] T d2b7cd739f0d4034b3411ba37301a08c P 571b1f82c62640baa5f92127434ff04e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ee346edbe3a4443bb23cccccd8b9f75 in term 0.
I20260929 18:32:41.123063 5150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:41.115461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:38545 (local address 127.28.207.67:55351)
0929 18:32:41.115909 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:41.115927 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:41.116022 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:41.116308 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:41.116312 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:41.116360 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:41.116603 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:41.116612 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:41.117946 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:41.117949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:41.119773 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:41.119785 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:41.120964 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:41.120974 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:41.121062 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:41.121086 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 18:32:41.122320 (+ 1234us) client_negotiation.cc:240] Negotiation successful
0929 18:32:41.122794 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":336,"spinlock_wait_cycles":5504,"thread_start_us":100,"threads_started":1}
I20260929 18:32:41.123616 5148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:41.115279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:38705 (local address 127.28.207.67:55133)
0929 18:32:41.115672 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:41.115692 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:41.115840 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:41.116308 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:41.116312 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:41.116359 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:32:41.116605 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:41.116612 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:41.117945 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:41.117949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:41.119774 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:41.119785 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:41.121591 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:41.121595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:41.121683 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:41.121707 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 18:32:41.121767 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 18:32:41.121842 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":220,"spinlock_wait_cycles":42368,"thread_start_us":118,"threads_started":1}
I20260929 18:32:41.124483 4722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2b7cd739f0d4034b3411ba37301a08c" candidate_uuid: "571b1f82c62640baa5f92127434ff04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb628a7dcc2441aa375002c139dcace" is_pre_election: true
I20260929 18:32:41.124774 4722 raft_consensus.cc:2454] T d2b7cd739f0d4034b3411ba37301a08c P 4eb628a7dcc2441aa375002c139dcace [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 571b1f82c62640baa5f92127434ff04e in term 0.
I20260929 18:32:41.125348 4933 leader_election.cc:302] T d2b7cd739f0d4034b3411ba37301a08c P 571b1f82c62640baa5f92127434ff04e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb628a7dcc2441aa375002c139dcace, 571b1f82c62640baa5f92127434ff04e; no voters:
I20260929 18:32:41.125766 5083 raft_consensus.cc:2781] T d2b7cd739f0d4034b3411ba37301a08c P 571b1f82c62640baa5f92127434ff04e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:32:41.125875 5083 raft_consensus.cc:482] T d2b7cd739f0d4034b3411ba37301a08c P 571b1f82c62640baa5f92127434ff04e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:32:41.125934 5083 raft_consensus.cc:3037] T d2b7cd739f0d4034b3411ba37301a08c P 571b1f82c62640baa5f92127434ff04e [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:41.126138 5086 raft_consensus.cc:2781] T d2b7cd739f0d4034b3411ba37301a08c P 4ee346edbe3a4443bb23cccccd8b9f75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:32:41.126226 5086 raft_consensus.cc:482] T d2b7cd739f0d4034b3411ba37301a08c P 4ee346edbe3a4443bb23cccccd8b9f75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:32:41.126266 5086 raft_consensus.cc:3037] T d2b7cd739f0d4034b3411ba37301a08c P 4ee346edbe3a4443bb23cccccd8b9f75 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:41.126308 4862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2b7cd739f0d4034b3411ba37301a08c" candidate_uuid: "571b1f82c62640baa5f92127434ff04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" is_pre_election: true
I20260929 18:32:41.127593 5086 raft_consensus.cc:504] T d2b7cd739f0d4034b3411ba37301a08c P 4ee346edbe3a4443bb23cccccd8b9f75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:41.127630 5083 raft_consensus.cc:504] T d2b7cd739f0d4034b3411ba37301a08c P 571b1f82c62640baa5f92127434ff04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:41.127926 4862 raft_consensus.cc:2379] T d2b7cd739f0d4034b3411ba37301a08c P 4ee346edbe3a4443bb23cccccd8b9f75 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 571b1f82c62640baa5f92127434ff04e in current term 1: Already voted for candidate 4ee346edbe3a4443bb23cccccd8b9f75 in this term.
I20260929 18:32:41.128067 5086 leader_election.cc:288] T d2b7cd739f0d4034b3411ba37301a08c P 4ee346edbe3a4443bb23cccccd8b9f75 [CANDIDATE]: Term 1 election: Requested vote from peers 4eb628a7dcc2441aa375002c139dcace (127.28.207.65:38705), 571b1f82c62640baa5f92127434ff04e (127.28.207.67:40275)
I20260929 18:32:41.128113 5083 leader_election.cc:288] T d2b7cd739f0d4034b3411ba37301a08c P 571b1f82c62640baa5f92127434ff04e [CANDIDATE]: Term 1 election: Requested vote from peers 4eb628a7dcc2441aa375002c139dcace (127.28.207.65:38705), 4ee346edbe3a4443bb23cccccd8b9f75 (127.28.207.66:38545)
I20260929 18:32:41.129137 5002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2b7cd739f0d4034b3411ba37301a08c" candidate_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "571b1f82c62640baa5f92127434ff04e"
I20260929 18:32:41.129139 4722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2b7cd739f0d4034b3411ba37301a08c" candidate_uuid: "571b1f82c62640baa5f92127434ff04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb628a7dcc2441aa375002c139dcace"
I20260929 18:32:41.129259 4722 raft_consensus.cc:3037] T d2b7cd739f0d4034b3411ba37301a08c P 4eb628a7dcc2441aa375002c139dcace [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:41.129318 5002 raft_consensus.cc:2379] T d2b7cd739f0d4034b3411ba37301a08c P 571b1f82c62640baa5f92127434ff04e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4ee346edbe3a4443bb23cccccd8b9f75 in current term 1: Already voted for candidate 571b1f82c62640baa5f92127434ff04e in this term.
I20260929 18:32:41.130697 4722 raft_consensus.cc:2454] T d2b7cd739f0d4034b3411ba37301a08c P 4eb628a7dcc2441aa375002c139dcace [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571b1f82c62640baa5f92127434ff04e in term 1.
I20260929 18:32:41.131268 4933 leader_election.cc:302] T d2b7cd739f0d4034b3411ba37301a08c P 571b1f82c62640baa5f92127434ff04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb628a7dcc2441aa375002c139dcace, 571b1f82c62640baa5f92127434ff04e; no voters:
I20260929 18:32:41.131393 4722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2b7cd739f0d4034b3411ba37301a08c" candidate_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb628a7dcc2441aa375002c139dcace"
I20260929 18:32:41.131470 5083 raft_consensus.cc:2781] T d2b7cd739f0d4034b3411ba37301a08c P 571b1f82c62640baa5f92127434ff04e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:32:41.131539 4862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2b7cd739f0d4034b3411ba37301a08c" candidate_uuid: "571b1f82c62640baa5f92127434ff04e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ee346edbe3a4443bb23cccccd8b9f75"
I20260929 18:32:41.131621 4722 raft_consensus.cc:2379] T d2b7cd739f0d4034b3411ba37301a08c P 4eb628a7dcc2441aa375002c139dcace [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4ee346edbe3a4443bb23cccccd8b9f75 in current term 1: Already voted for candidate 571b1f82c62640baa5f92127434ff04e in this term.
I20260929 18:32:41.131660 5083 raft_consensus.cc:686] T d2b7cd739f0d4034b3411ba37301a08c P 571b1f82c62640baa5f92127434ff04e [term 1 LEADER]: Becoming Leader. State: Replica: 571b1f82c62640baa5f92127434ff04e, State: Running, Role: LEADER
I20260929 18:32:41.131699 4862 raft_consensus.cc:2379] T d2b7cd739f0d4034b3411ba37301a08c P 4ee346edbe3a4443bb23cccccd8b9f75 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 571b1f82c62640baa5f92127434ff04e in current term 1: Already voted for candidate 4ee346edbe3a4443bb23cccccd8b9f75 in this term.
I20260929 18:32:41.131943 5083 consensus_queue.cc:237] T d2b7cd739f0d4034b3411ba37301a08c P 571b1f82c62640baa5f92127434ff04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } }
I20260929 18:32:41.131996 4793 leader_election.cc:302] T d2b7cd739f0d4034b3411ba37301a08c P 4ee346edbe3a4443bb23cccccd8b9f75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4ee346edbe3a4443bb23cccccd8b9f75; no voters: 4eb628a7dcc2441aa375002c139dcace, 571b1f82c62640baa5f92127434ff04e
I20260929 18:32:41.133317 5086 raft_consensus.cc:2726] T d2b7cd739f0d4034b3411ba37301a08c P 4ee346edbe3a4443bb23cccccd8b9f75 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260929 18:32:41.135008 4562 catalog_manager.cc:5495] T d2b7cd739f0d4034b3411ba37301a08c P 571b1f82c62640baa5f92127434ff04e reported cstate change: term changed from 0 to 1, leader changed from <none> to 571b1f82c62640baa5f92127434ff04e (127.28.207.67). New cstate: current_term: 1 leader_uuid: "571b1f82c62640baa5f92127434ff04e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "571b1f82c62640baa5f92127434ff04e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 40275 } health_report { overall_health: HEALTHY } } }
I20260929 18:32:41.211004 5153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:41.206632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33902 (local address 127.28.207.67:40275)
0929 18:32:41.207026 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:41.207035 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:41.207053 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:32:41.207127 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:41.207129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:41.207186 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:41.207352 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:32:41.207640 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:41.208200 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:41.209294 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:41.209579 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:41.209672 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:41.210004 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:32:41.210519 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:32:41.210621 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:41.210732 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 18:32:41.210783 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":125,"threads_started":1}
I20260929 18:32:41.224440 5073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:41.220255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:56945 (local address 127.28.207.66:38545)
0929 18:32:41.220375 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:41.220379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:41.220851 (+ 472us) server_negotiation.cc:407] Connection header received
0929 18:32:41.220985 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:41.220987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:41.221032 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:41.221127 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:32:41.221534 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:41.222064 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:41.223119 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:41.223328 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:41.223425 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:41.223902 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:32:41.223967 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:32:41.224030 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:41.224273 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 18:32:41.224314 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 18:32:41.225003 5156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:41.220171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:38545 (local address 127.28.207.67:56945)
0929 18:32:41.220748 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:41.220765 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:41.220890 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:41.221199 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:41.221201 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:41.221209 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:32:41.221343 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:41.221348 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:41.222237 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:41.222240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:41.222990 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:41.222999 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:41.223472 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:41.223475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:41.223654 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:41.223752 (+ 98us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:32:41.224097 (+ 345us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:32:41.224104 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 18:32:41.224265 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 18:32:41.224350 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":180,"threads_started":1}
I20260929 18:32:41.232527 4722 raft_consensus.cc:1261] T d2b7cd739f0d4034b3411ba37301a08c P 4eb628a7dcc2441aa375002c139dcace [term 1 FOLLOWER]: Refusing update from remote peer 571b1f82c62640baa5f92127434ff04e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:32:41.232754 4862 raft_consensus.cc:1261] T d2b7cd739f0d4034b3411ba37301a08c P 4ee346edbe3a4443bb23cccccd8b9f75 [term 1 FOLLOWER]: Refusing update from remote peer 571b1f82c62640baa5f92127434ff04e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:32:41.233006 5083 consensus_queue.cc:1035] T d2b7cd739f0d4034b3411ba37301a08c P 571b1f82c62640baa5f92127434ff04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb628a7dcc2441aa375002c139dcace" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 38705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:32:41.233258 5151 consensus_queue.cc:1035] T d2b7cd739f0d4034b3411ba37301a08c P 571b1f82c62640baa5f92127434ff04e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ee346edbe3a4443bb23cccccd8b9f75" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 38545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:32:41.261835 5161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:41.255577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:40275 (local address 127.28.207.66:56589)
0929 18:32:41.256284 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:41.256301 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:41.256518 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:41.256878 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:41.256881 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:41.256890 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 18:32:41.257443 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:41.257451 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:41.258635 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:41.258638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:41.259791 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:41.259801 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:41.260767 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:41.260772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:41.260965 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:41.261008 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 18:32:41.261430 (+ 422us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 18:32:41.261438 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 18:32:41.261559 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 18:32:41.261684 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":130,"threads_started":1}
I20260929 18:32:41.261857 5153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:41.255734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:56589 (local address 127.28.207.67:40275)
0929 18:32:41.255924 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:41.255930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:41.256431 (+ 501us) server_negotiation.cc:407] Connection header received
0929 18:32:41.256669 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:41.256673 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:41.256746 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:41.256847 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:32:41.257589 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:41.258423 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:41.259926 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:41.260314 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:41.260773 (+ 459us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:41.261177 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:32:41.261285 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:32:41.261378 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:41.261621 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 18:32:41.261712 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 18:32:41.287155 4565 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:32:41.287395 4565 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:32:41.290230 4565 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf94f8dce239472093ae97936b750185: Sending DeleteTablet for 3 replicas of tablet d2b7cd739f0d4034b3411ba37301a08c
I20260929 18:32:41.291074 4842 tablet_service.cc:1514] Processing DeleteTablet for tablet d2b7cd739f0d4034b3411ba37301a08c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:41 UTC) from {username='oryx', principal='oryx/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:33144
I20260929 18:32:41.291085 4702 tablet_service.cc:1514] Processing DeleteTablet for tablet d2b7cd739f0d4034b3411ba37301a08c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:41 UTC) from {username='oryx', principal='oryx/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:40364
I20260929 18:32:41.291191 4982 tablet_service.cc:1514] Processing DeleteTablet for tablet d2b7cd739f0d4034b3411ba37301a08c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:32:41 UTC) from {username='oryx', principal='oryx/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:33892
I20260929 18:32:41.291350 5099 tablet_replica.cc:330] T d2b7cd739f0d4034b3411ba37301a08c P 4eb628a7dcc2441aa375002c139dcace: stopping tablet replica
I20260929 18:32:41.291380 5101 tablet_replica.cc:330] T d2b7cd739f0d4034b3411ba37301a08c P 4ee346edbe3a4443bb23cccccd8b9f75: stopping tablet replica
I20260929 18:32:41.291496 5099 raft_consensus.cc:2229] T d2b7cd739f0d4034b3411ba37301a08c P 4eb628a7dcc2441aa375002c139dcace [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:32:41.291548 5101 raft_consensus.cc:2229] T d2b7cd739f0d4034b3411ba37301a08c P 4ee346edbe3a4443bb23cccccd8b9f75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:32:41.291635 5099 raft_consensus.cc:2258] T d2b7cd739f0d4034b3411ba37301a08c P 4eb628a7dcc2441aa375002c139dcace [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:41.291703 5101 raft_consensus.cc:2258] T d2b7cd739f0d4034b3411ba37301a08c P 4ee346edbe3a4443bb23cccccd8b9f75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:41.291906 5100 tablet_replica.cc:330] T d2b7cd739f0d4034b3411ba37301a08c P 571b1f82c62640baa5f92127434ff04e: stopping tablet replica
I20260929 18:32:41.292076 5100 raft_consensus.cc:2229] T d2b7cd739f0d4034b3411ba37301a08c P 571b1f82c62640baa5f92127434ff04e [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:32:41.292471 5100 raft_consensus.cc:2258] T d2b7cd739f0d4034b3411ba37301a08c P 571b1f82c62640baa5f92127434ff04e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:32:41.292847 5099 ts_tablet_manager.cc:1905] T d2b7cd739f0d4034b3411ba37301a08c P 4eb628a7dcc2441aa375002c139dcace: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:32:41.293751 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 4637
I20260929 18:32:41.293977 5100 ts_tablet_manager.cc:1905] T d2b7cd739f0d4034b3411ba37301a08c P 571b1f82c62640baa5f92127434ff04e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:32:41.296715 5100 ts_tablet_manager.cc:1918] T d2b7cd739f0d4034b3411ba37301a08c P 571b1f82c62640baa5f92127434ff04e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:32:41.296814 5100 log.cc:1196] T d2b7cd739f0d4034b3411ba37301a08c P 571b1f82c62640baa5f92127434ff04e: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-2/wal/wals/d2b7cd739f0d4034b3411ba37301a08c
I20260929 18:32:41.297083 5100 ts_tablet_manager.cc:1939] T d2b7cd739f0d4034b3411ba37301a08c P 571b1f82c62640baa5f92127434ff04e: Deleting consensus metadata
I20260929 18:32:41.297573 5101 ts_tablet_manager.cc:1905] T d2b7cd739f0d4034b3411ba37301a08c P 4ee346edbe3a4443bb23cccccd8b9f75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:32:41.300072 5101 ts_tablet_manager.cc:1918] T d2b7cd739f0d4034b3411ba37301a08c P 4ee346edbe3a4443bb23cccccd8b9f75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 18:32:41.300182 5101 log.cc:1196] T d2b7cd739f0d4034b3411ba37301a08c P 4ee346edbe3a4443bb23cccccd8b9f75: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790706718260899-29501-0/minicluster-data/ts-1/wal/wals/d2b7cd739f0d4034b3411ba37301a08c
I20260929 18:32:41.300387 5101 ts_tablet_manager.cc:1939] T d2b7cd739f0d4034b3411ba37301a08c P 4ee346edbe3a4443bb23cccccd8b9f75: Deleting consensus metadata
I20260929 18:32:41.301889 4552 catalog_manager.cc:4841] TS 571b1f82c62640baa5f92127434ff04e (127.28.207.67:40275): tablet d2b7cd739f0d4034b3411ba37301a08c (table test-table [id=bd5a9abf629b4f6a8ea268e3219a98b2]) successfully deleted
I20260929 18:32:41.301990 4550 catalog_manager.cc:4841] TS 4ee346edbe3a4443bb23cccccd8b9f75 (127.28.207.66:38545): tablet d2b7cd739f0d4034b3411ba37301a08c (table test-table [id=bd5a9abf629b4f6a8ea268e3219a98b2]) successfully deleted
W20260929 18:32:41.305236 4549 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.207.65:38705 (error 108)
I20260929 18:32:41.305428 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 4777
I20260929 18:32:41.308845 5165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:41.308341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:38705 (local address 127.0.0.1:40370)
0929 18:32:41.308616 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:41.308712 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.65:38705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":85,"threads_started":1}
W20260929 18:32:41.309224 4549 catalog_manager.cc:4584] TS 4eb628a7dcc2441aa375002c139dcace (127.28.207.65:38705): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d2b7cd739f0d4034b3411ba37301a08c: Network error: Client connection negotiation failed: client connection to 127.28.207.65:38705: connect: Connection refused (error 111)
I20260929 18:32:41.316709 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 4917
I20260929 18:32:41.324285 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 4533
2026-09-29T18:32:41Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3577 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:32:41 dist-test-slave-qcpv krb5kdc[5168](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:32:41 dist-test-slave-qcpv krb5kdc[5168](info): set up 2 sockets
Sep 29 18:32:41 dist-test-slave-qcpv krb5kdc[5168](info): commencing operation
krb5kdc: starting...
W20260929 18:32:42.395897 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:32:42 dist-test-slave-qcpv krb5kdc[5168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:32:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:32:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.207.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
I20260929 18:32:42.547487 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:43893
--webserver_interface=127.28.207.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:35165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:43893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:42.665670 5181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:42.665985 5181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:42.666029 5181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:42.670243 5181 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:32:42.670328 5181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:42.670349 5181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:42.670367 5181 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:32:42.670385 5181 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:32:42.676095 5181 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:35165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:43893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:43893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:42.677259 5181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:42.678306 5181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:42.688462 5189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:42.688480 5187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:42.688555 5181 server_base.cc:1034] running on GCE node
W20260929 18:32:42.688480 5186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:42.689601 5181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:42.690914 5181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:42.692123 5181 hybrid_clock.cc:648] HybridClock initialized: now 1790706762692110 us; error 31 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T18:32:42Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1368 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:32:42 dist-test-slave-qcpv krb5kdc[5194](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:32:42 dist-test-slave-qcpv krb5kdc[5194](info): set up 2 sockets
Sep 29 18:32:42 dist-test-slave-qcpv krb5kdc[5194](info): commencing operation
krb5kdc: starting...
W20260929 18:32:43.796353 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.008s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:32:43 dist-test-slave-qcpv krb5kdc[5194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:32:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:32:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.207.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
I20260929 18:32:43.947634 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:45245
--webserver_interface=127.28.207.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:35539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:45245
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:44.068842 5207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:44.069198 5207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:44.069273 5207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:44.073449 5207 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:32:44.073536 5207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:44.073575 5207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:44.073629 5207 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:32:44.073678 5207 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:32:44.079815 5207 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:35539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:45245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:45245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:44.081010 5207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:44.082069 5207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:44.092149 5215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:44.092183 5213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:44.092363 5207 server_base.cc:1034] running on GCE node
W20260929 18:32:44.092134 5212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:44.093364 5207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:44.094992 5207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:44.096222 5207 hybrid_clock.cc:648] HybridClock initialized: now 1790706764096197 us; error 51 us; skew 500 ppm
Sep 29 18:32:44 dist-test-slave-qcpv krb5kdc[5194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706764, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:44.100251 5207 init.cc:377] Logged in from keytab as kudu/127.28.207.126@KRBTEST.COM (short username kudu)
I20260929 18:32:44.101990 5207 webserver.cc:460] Webserver started at http://127.28.207.126:44895/ using document root <none> and password file <none>
I20260929 18:32:44.103070 5207 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:44.103188 5207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:44.103480 5207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:44.106400 5207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "c605752334ab4ed897fd456b3e0e4462"
format_stamp: "Formatted at 2026-09-29 18:32:44 on dist-test-slave-qcpv"
server_key: "db4c71accb6c6247f0fabf398ee8dd06"
server_key_iv: "9cce9dd7751ba549d03306115ee16ae2"
server_key_version: "encryptionkey@0"
I20260929 18:32:44.107196 5207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "c605752334ab4ed897fd456b3e0e4462"
format_stamp: "Formatted at 2026-09-29 18:32:44 on dist-test-slave-qcpv"
server_key: "db4c71accb6c6247f0fabf398ee8dd06"
server_key_iv: "9cce9dd7751ba549d03306115ee16ae2"
server_key_version: "encryptionkey@0"
I20260929 18:32:44.112437 5207 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260929 18:32:44.116580 5222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:44.118340 5207 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:32:44.118553 5207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "c605752334ab4ed897fd456b3e0e4462"
format_stamp: "Formatted at 2026-09-29 18:32:44 on dist-test-slave-qcpv"
server_key: "db4c71accb6c6247f0fabf398ee8dd06"
server_key_iv: "9cce9dd7751ba549d03306115ee16ae2"
server_key_version: "encryptionkey@0"
I20260929 18:32:44.118727 5207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:44.143750 5207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:44.148275 5207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:44.148524 5207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:44.156390 5207 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:45245
I20260929 18:32:44.156627 5274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:45245 every 8 connection(s)
I20260929 18:32:44.157501 5207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:32:44.160528 5275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:32:44.166811 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 5207
I20260929 18:32:44.167006 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:32:44.167285 29501 external_mini_cluster.cc:1428] Setting key f1665b86e146486ddad09513a4c2f72c
I20260929 18:32:44.168603 5275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462: Bootstrap starting.
I20260929 18:32:44.172248 5275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462: Neither blocks nor log segments found. Creating new log.
I20260929 18:32:44.173394 5275 log.cc:824] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:44.176398 5275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462: No bootstrap required, opened a new log
I20260929 18:32:44.181015 5275 raft_consensus.cc:348] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c605752334ab4ed897fd456b3e0e4462" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 45245 } }
I20260929 18:32:44.181341 5275 raft_consensus.cc:374] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:32:44.181391 5275 raft_consensus.cc:729] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c605752334ab4ed897fd456b3e0e4462, State: Initialized, Role: FOLLOWER
I20260929 18:32:44.181921 5275 consensus_queue.cc:260] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c605752334ab4ed897fd456b3e0e4462" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 45245 } }
I20260929 18:32:44.182060 5275 raft_consensus.cc:388] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:32:44.182109 5275 raft_consensus.cc:482] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:32:44.182195 5275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:32:44.183517 5275 raft_consensus.cc:504] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c605752334ab4ed897fd456b3e0e4462" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 45245 } }
I20260929 18:32:44.183923 5275 leader_election.cc:302] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c605752334ab4ed897fd456b3e0e4462; no voters:
I20260929 18:32:44.184314 5275 leader_election.cc:288] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:32:44.184602 5280 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:32:44.184975 5280 raft_consensus.cc:686] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462 [term 1 LEADER]: Becoming Leader. State: Replica: c605752334ab4ed897fd456b3e0e4462, State: Running, Role: LEADER
I20260929 18:32:44.185424 5280 consensus_queue.cc:237] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c605752334ab4ed897fd456b3e0e4462" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 45245 } }
I20260929 18:32:44.186117 5275 sys_catalog.cc:564] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462 [sys.catalog]: configured and running, proceeding with master startup.
Sep 29 18:32:44 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:32:44.188659 5282 sys_catalog.cc:455] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c605752334ab4ed897fd456b3e0e4462. Latest consensus state: current_term: 1 leader_uuid: "c605752334ab4ed897fd456b3e0e4462" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c605752334ab4ed897fd456b3e0e4462" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 45245 } } }
I20260929 18:32:44.188861 5282 sys_catalog.cc:458] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462 [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:44.189198 5281 sys_catalog.cc:455] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c605752334ab4ed897fd456b3e0e4462" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c605752334ab4ed897fd456b3e0e4462" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 45245 } } }
I20260929 18:32:44.189302 5281 sys_catalog.cc:458] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462 [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:44.190018 5294 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:32:44.193899 5294 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:32:44.196698 5278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:44.169159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43458 (local address 127.28.207.126:45245)
0929 18:32:44.169725 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:44.169750 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:44.169808 (+ 58us) server_negotiation.cc:407] Connection header received
0929 18:32:44.171013 (+ 1205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:44.171043 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:44.171602 (+ 559us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:44.172102 (+ 500us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:44.173073 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:44.174627 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:44.175760 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:44.176561 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:44.176772 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:44.189218 (+ 12446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:44.189228 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:44.189251 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:44.189299 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:44.191872 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:44.192784 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:44.192790 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:44.192796 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:44.192969 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:44.193365 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:44.193368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:44.193370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:44.193954 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:44.194176 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:44.194798 (+ 622us) server_negotiation.cc:299] Negotiation successful
0929 18:32:44.195378 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":151,"threads_started":1}
I20260929 18:32:44.200598 5294 catalog_manager.cc:1347] Generated new cluster ID: 09ad7be156324e94bb17e697afda34ff
I20260929 18:32:44.200744 5294 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:32:44.206497 5294 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:32:44.207748 5294 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:32:44.222894 5294 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462: Generated new TSK 0
I20260929 18:32:44.223778 5294 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.207.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab.
Entry for principal kudu/127.28.207.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab.
I20260929 18:32:44.263132 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.65:0
--local_ip_for_outbound_sockets=127.28.207.65
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:45245
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:35539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:44.410225 5301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:44.410790 5301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:44.410945 5301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:44.417428 5301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:44.417524 5301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:44.417722 5301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.65
I20260929 18:32:44.424546 5301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:35539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:45245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.65
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:44.425936 5301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:44.427266 5301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:44.440408 5307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:44.440482 5309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:44.440600 5301 server_base.cc:1034] running on GCE node
W20260929 18:32:44.440500 5306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:44.441334 5301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:44.442150 5301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:44.443439 5301 hybrid_clock.cc:648] HybridClock initialized: now 1790706764443383 us; error 63 us; skew 500 ppm
Sep 29 18:32:44 dist-test-slave-qcpv krb5kdc[5194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706764, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:44.446974 5301 init.cc:377] Logged in from keytab as kudu/127.28.207.65@KRBTEST.COM (short username kudu)
I20260929 18:32:44.448225 5301 webserver.cc:460] Webserver started at http://127.28.207.65:32785/ using document root <none> and password file <none>
I20260929 18:32:44.448959 5301 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:44.449056 5301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:44.449328 5301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:44.452081 5301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-0/data/instance:
uuid: "24d6415b001f4ad9b42c93ca0b0f296e"
format_stamp: "Formatted at 2026-09-29 18:32:44 on dist-test-slave-qcpv"
server_key: "0532ea416869e1e2e5420dd32567e518"
server_key_iv: "926f211cdaa915aca95316893431a058"
server_key_version: "encryptionkey@0"
I20260929 18:32:44.452766 5301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance:
uuid: "24d6415b001f4ad9b42c93ca0b0f296e"
format_stamp: "Formatted at 2026-09-29 18:32:44 on dist-test-slave-qcpv"
server_key: "0532ea416869e1e2e5420dd32567e518"
server_key_iv: "926f211cdaa915aca95316893431a058"
server_key_version: "encryptionkey@0"
I20260929 18:32:44.457685 5301 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 18:32:44.462172 5316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:44.464052 5301 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:32:44.464265 5301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-0/wal
uuid: "24d6415b001f4ad9b42c93ca0b0f296e"
format_stamp: "Formatted at 2026-09-29 18:32:44 on dist-test-slave-qcpv"
server_key: "0532ea416869e1e2e5420dd32567e518"
server_key_iv: "926f211cdaa915aca95316893431a058"
server_key_version: "encryptionkey@0"
I20260929 18:32:44.464430 5301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:44.493330 5301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:44.497632 5301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:44.497892 5301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:44.498680 5301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:32:44.500052 5301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:44.500156 5301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:44.500234 5301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:44.500273 5301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:44.512624 5301 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.65:34175
I20260929 18:32:44.512717 5429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.65:34175 every 8 connection(s)
I20260929 18:32:44.513834 5301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
I20260929 18:32:44.522607 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 5301
I20260929 18:32:44.522743 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance
I20260929 18:32:44.523131 29501 external_mini_cluster.cc:1428] Setting key 2f18c06b4243cbc8cf6827f90f4dcf32
Sep 29 18:32:44 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706764, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:32:44.530941 5278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:44.516260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:57235 (local address 127.28.207.126:45245)
0929 18:32:44.516440 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:44.516444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:44.517645 (+ 1201us) server_negotiation.cc:407] Connection header received
0929 18:32:44.519187 (+ 1542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:44.519191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:44.519253 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:44.519366 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:44.520629 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:44.521170 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:44.522516 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:44.522892 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:44.522999 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:44.526241 (+ 3242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:44.526245 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:44.526248 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:44.526276 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:44.528201 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:44.528904 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:44.528910 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:44.528912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:44.529007 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:44.529433 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:44.529437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:44.529438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:44.529687 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:44.529807 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:44.530489 (+ 682us) server_negotiation.cc:299] Negotiation successful
0929 18:32:44.530678 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 18:32:44.532109 5432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:44.516858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.65:57235)
0929 18:32:44.517471 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:44.517525 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:44.518938 (+ 1413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:44.519553 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:44.519560 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:44.519970 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:44.520484 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:44.520497 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:44.521292 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:44.521296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:44.522291 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:44.522332 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:44.523075 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:44.523079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:44.523284 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:44.524017 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:44.524040 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:44.525962 (+ 1922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:44.528371 (+ 2409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:44.528379 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:44.528406 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:44.528780 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:44.529131 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:44.529135 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:44.529137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:44.529318 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:44.529804 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:44.529815 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:44.530175 (+ 360us) client_negotiation.cc:769] Sending connection context
0929 18:32:44.530449 (+ 274us) client_negotiation.cc:240] Negotiation successful
0929 18:32:44.530939 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":172,"threads_started":1}
WARNING: no policy specified for kudu/127.28.207.66@KRBTEST.COM; defaulting to no policy
I20260929 18:32:44.533906 5430 heartbeater.cc:347] Connected to a master server at 127.28.207.126:45245
I20260929 18:32:44.534226 5430 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:44.534847 5430 heartbeater.cc:511] Master 127.28.207.126:45245 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.66@KRBTEST.COM" created.
I20260929 18:32:44.536923 5234 ts_manager.cc:194] Registered new tserver with Master: 24d6415b001f4ad9b42c93ca0b0f296e (127.28.207.65:34175)
I20260929 18:32:44.538813 5234 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.65@KRBTEST.COM'} at 127.28.207.65:57235
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab.
Entry for principal kudu/127.28.207.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab.
I20260929 18:32:44.555480 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.66:0
--local_ip_for_outbound_sockets=127.28.207.66
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:45245
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:35539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:44.676725 5435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:44.676987 5435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:44.677031 5435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:44.681083 5435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:44.681166 5435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:44.681229 5435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.66
I20260929 18:32:44.686818 5435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:35539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:45245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.66
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:44.688203 5435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:44.689467 5435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:44.700279 5440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:44.700307 5443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:44.700279 5441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:44.700434 5435 server_base.cc:1034] running on GCE node
I20260929 18:32:44.701184 5435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:44.702286 5435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:44.703507 5435 hybrid_clock.cc:648] HybridClock initialized: now 1790706764703459 us; error 69 us; skew 500 ppm
Sep 29 18:32:44 dist-test-slave-qcpv krb5kdc[5194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706764, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:44.707075 5435 init.cc:377] Logged in from keytab as kudu/127.28.207.66@KRBTEST.COM (short username kudu)
I20260929 18:32:44.708295 5435 webserver.cc:460] Webserver started at http://127.28.207.66:45511/ using document root <none> and password file <none>
I20260929 18:32:44.709025 5435 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:44.709124 5435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:44.709393 5435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:44.712214 5435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-1/data/instance:
uuid: "812d22c4e03e4d649b68018f12253819"
format_stamp: "Formatted at 2026-09-29 18:32:44 on dist-test-slave-qcpv"
server_key: "3a7819982c98d46add89c81295e97a50"
server_key_iv: "9b78bd37cd9fe4ddde8a9ee83cf3bd28"
server_key_version: "encryptionkey@0"
I20260929 18:32:44.712910 5435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance:
uuid: "812d22c4e03e4d649b68018f12253819"
format_stamp: "Formatted at 2026-09-29 18:32:44 on dist-test-slave-qcpv"
server_key: "3a7819982c98d46add89c81295e97a50"
server_key_iv: "9b78bd37cd9fe4ddde8a9ee83cf3bd28"
server_key_version: "encryptionkey@0"
I20260929 18:32:44.717717 5435 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 18:32:44.721342 5450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:44.723373 5435 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:32:44.723558 5435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-1/wal
uuid: "812d22c4e03e4d649b68018f12253819"
format_stamp: "Formatted at 2026-09-29 18:32:44 on dist-test-slave-qcpv"
server_key: "3a7819982c98d46add89c81295e97a50"
server_key_iv: "9b78bd37cd9fe4ddde8a9ee83cf3bd28"
server_key_version: "encryptionkey@0"
I20260929 18:32:44.723711 5435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:44.742794 5435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:44.747128 5435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:44.747370 5435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:44.748116 5435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:32:44.749382 5435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:44.749470 5435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:44.749558 5435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:44.749606 5435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:44.761631 5435 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.66:34743
I20260929 18:32:44.761669 5563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.66:34743 every 8 connection(s)
I20260929 18:32:44.762813 5435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
I20260929 18:32:44.763754 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 5435
I20260929 18:32:44.763887 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance
I20260929 18:32:44.764222 29501 external_mini_cluster.cc:1428] Setting key 105233b206b2fe40f7a3e238bfc3507a
Sep 29 18:32:44 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706764, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.207.67@KRBTEST.COM; defaulting to no policy
I20260929 18:32:44.779467 5278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:44.765373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:45487 (local address 127.28.207.126:45245)
0929 18:32:44.765541 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:44.765545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:44.767136 (+ 1591us) server_negotiation.cc:407] Connection header received
0929 18:32:44.768674 (+ 1538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:44.768677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:44.768739 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:44.768845 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:44.770152 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:44.770689 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:44.771897 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:44.772115 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:44.772218 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:44.775083 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:44.775087 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:44.775089 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:44.775114 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:44.776649 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:44.777412 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:44.777415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:44.777416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:44.777494 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:44.777998 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:44.778001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:44.778002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:44.778186 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:44.778281 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:44.779216 (+ 935us) server_negotiation.cc:299] Negotiation successful
0929 18:32:44.779331 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.67@KRBTEST.COM" created.
I20260929 18:32:44.780766 5566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:44.766034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.66:45487)
0929 18:32:44.766968 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:44.767022 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:44.768421 (+ 1399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:44.769028 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:44.769036 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:44.769487 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:44.770002 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:44.770015 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:44.770809 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:44.770813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:44.771770 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:44.771779 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:44.772275 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:44.772279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:44.772447 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:44.773366 (+ 919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:44.773381 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:44.774919 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:44.776845 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:44.776858 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:44.776887 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:44.777271 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:44.777625 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:44.777628 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:44.777630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:44.777881 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:44.778364 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:44.778374 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:44.778719 (+ 345us) client_negotiation.cc:769] Sending connection context
0929 18:32:44.779087 (+ 368us) client_negotiation.cc:240] Negotiation successful
0929 18:32:44.779448 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":390,"threads_started":1}
I20260929 18:32:44.782259 5564 heartbeater.cc:347] Connected to a master server at 127.28.207.126:45245
I20260929 18:32:44.782604 5564 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:44.783636 5564 heartbeater.cc:511] Master 127.28.207.126:45245 requested a full tablet report, sending...
I20260929 18:32:44.785236 5234 ts_manager.cc:194] Registered new tserver with Master: 812d22c4e03e4d649b68018f12253819 (127.28.207.66:34743)
I20260929 18:32:44.786572 5234 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.66@KRBTEST.COM'} at 127.28.207.66:45487
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab.
Entry for principal kudu/127.28.207.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab.
I20260929 18:32:44.804033 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.67:0
--local_ip_for_outbound_sockets=127.28.207.67
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:45245
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:35539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:44.953157 5569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:44.953541 5569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:44.953593 5569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:44.958683 5569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:44.958792 5569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:44.958889 5569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.67
I20260929 18:32:44.967109 5569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:35539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:45245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.67
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:44.968729 5569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:44.970117 5569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:44.983583 5577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:44.983691 5574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:44.983817 5569 server_base.cc:1034] running on GCE node
W20260929 18:32:44.984314 5575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:44.984987 5569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:44.985879 5569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:44.987157 5569 hybrid_clock.cc:648] HybridClock initialized: now 1790706764987117 us; error 58 us; skew 500 ppm
Sep 29 18:32:44 dist-test-slave-qcpv krb5kdc[5194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706764, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:44.991266 5569 init.cc:377] Logged in from keytab as kudu/127.28.207.67@KRBTEST.COM (short username kudu)
I20260929 18:32:44.992887 5569 webserver.cc:460] Webserver started at http://127.28.207.67:37933/ using document root <none> and password file <none>
I20260929 18:32:44.993672 5569 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:44.993758 5569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:44.994028 5569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:32:44.997110 5569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-2/data/instance:
uuid: "9c06d48ff10c40aab898a92bbc58226c"
format_stamp: "Formatted at 2026-09-29 18:32:44 on dist-test-slave-qcpv"
server_key: "f6d63fa3293c918ad12ae6196c1b1e11"
server_key_iv: "6a8c6660af76a50e720ffc35c05e1fd1"
server_key_version: "encryptionkey@0"
I20260929 18:32:44.997876 5569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance:
uuid: "9c06d48ff10c40aab898a92bbc58226c"
format_stamp: "Formatted at 2026-09-29 18:32:44 on dist-test-slave-qcpv"
server_key: "f6d63fa3293c918ad12ae6196c1b1e11"
server_key_iv: "6a8c6660af76a50e720ffc35c05e1fd1"
server_key_version: "encryptionkey@0"
I20260929 18:32:45.003247 5569 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 18:32:45.007510 5584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:45.009600 5569 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 18:32:45.009825 5569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-2/wal
uuid: "9c06d48ff10c40aab898a92bbc58226c"
format_stamp: "Formatted at 2026-09-29 18:32:44 on dist-test-slave-qcpv"
server_key: "f6d63fa3293c918ad12ae6196c1b1e11"
server_key_iv: "6a8c6660af76a50e720ffc35c05e1fd1"
server_key_version: "encryptionkey@0"
I20260929 18:32:45.010003 5569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:45.026324 5569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:45.031389 5569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:45.031665 5569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:45.032603 5569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:32:45.033738 5569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:45.033823 5569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:45.033900 5569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:45.033965 5569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:45.047672 5569 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.67:39471
I20260929 18:32:45.047850 5697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.67:39471 every 8 connection(s)
I20260929 18:32:45.048945 5569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
I20260929 18:32:45.053725 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 5569
I20260929 18:32:45.053862 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance
I20260929 18:32:45.054201 29501 external_mini_cluster.cc:1428] Setting key dcfc15890316bba0fb00cc334631343b
Sep 29 18:32:45 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706764, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:32:45.065876 5278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:45.051852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:46931 (local address 127.28.207.126:45245)
0929 18:32:45.052069 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:45.052073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:45.053330 (+ 1257us) server_negotiation.cc:407] Connection header received
0929 18:32:45.055239 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:45.055242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:45.055303 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:45.055399 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:45.056834 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:45.057380 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:45.058576 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:45.058787 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:45.058932 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:45.061974 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:45.061977 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:45.061980 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:45.062009 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:45.063749 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:45.064323 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:45.064327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:45.064330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:45.064406 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:45.064752 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:45.064757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:45.064758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:45.064951 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:45.065048 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:45.065570 (+ 522us) server_negotiation.cc:299] Negotiation successful
0929 18:32:45.065707 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260929 18:32:45.066817 5700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:45.052341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.67:46931)
0929 18:32:45.053091 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:45.053155 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:45.054936 (+ 1781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:45.055639 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:45.055650 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:45.056142 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:45.056685 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:45.056698 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:45.057508 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:45.057511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:45.058430 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:45.058439 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:45.058988 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:45.058992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:45.059175 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:45.059918 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:45.059938 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:45.061729 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:32:45.063905 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:45.063912 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:45.063930 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:45.064219 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:45.064506 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:32:45.064509 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:32:45.064511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:32:45.064660 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:32:45.065066 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:32:45.065074 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:32:45.065332 (+ 258us) client_negotiation.cc:769] Sending connection context
0929 18:32:45.065524 (+ 192us) client_negotiation.cc:240] Negotiation successful
0929 18:32:45.065871 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":264,"threads_started":1}
I20260929 18:32:45.069159 5698 heartbeater.cc:347] Connected to a master server at 127.28.207.126:45245
I20260929 18:32:45.069526 5698 heartbeater.cc:464] Registering TS with master...
I20260929 18:32:45.070300 5698 heartbeater.cc:511] Master 127.28.207.126:45245 requested a full tablet report, sending...
I20260929 18:32:45.071841 5234 ts_manager.cc:194] Registered new tserver with Master: 9c06d48ff10c40aab898a92bbc58226c (127.28.207.67:39471)
I20260929 18:32:45.072535 5234 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.67@KRBTEST.COM'} at 127.28.207.67:46931
I20260929 18:32:45.082469 29501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.28.207.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.207.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.207.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.65.keytab.
Entry for principal oryx/127.28.207.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.65.keytab.
WARNING: no policy specified for oryx/127.28.207.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.207.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.207.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.66.keytab.
Entry for principal oryx/127.28.207.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.66.keytab.
WARNING: no policy specified for oryx/127.28.207.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.28.207.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.28.207.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.67.keytab.
Entry for principal oryx/127.28.207.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.67.keytab.
I20260929 18:32:45.178437 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 5301
I20260929 18:32:45.184974 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 5435
I20260929 18:32:45.191222 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 5569
I20260929 18:32:45.197794 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 5207
I20260929 18:32:45.205169 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:45245
--webserver_interface=127.28.207.126
--webserver_port=44895
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:35539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:45245
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:32:45.346211 5707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:45.346671 5707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:45.346761 5707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:45.351349 5707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:32:45.351485 5707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:45.351521 5707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:45.351562 5707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:32:45.351606 5707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:32:45.358762 5707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:35539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:45245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:45245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=44895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:45.360150 5707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:45.361581 5707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:45.373282 5712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:45.373307 5715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:45.373526 5713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:45.373749 5707 server_base.cc:1034] running on GCE node
I20260929 18:32:45.374598 5707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:45.376137 5707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:45.377350 5707 hybrid_clock.cc:648] HybridClock initialized: now 1790706765377333 us; error 41 us; skew 500 ppm
Sep 29 18:32:45 dist-test-slave-qcpv krb5kdc[5194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706765, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:45.381532 5707 init.cc:377] Logged in from keytab as kudu/127.28.207.126@KRBTEST.COM (short username kudu)
I20260929 18:32:45.382802 5707 webserver.cc:460] Webserver started at http://127.28.207.126:44895/ using document root <none> and password file <none>
I20260929 18:32:45.383807 5707 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:45.383912 5707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:45.389019 5707 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:32:45.392112 5722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:45.393975 5707 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:32:45.394135 5707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "c605752334ab4ed897fd456b3e0e4462"
format_stamp: "Formatted at 2026-09-29 18:32:44 on dist-test-slave-qcpv"
server_key: "db4c71accb6c6247f0fabf398ee8dd06"
server_key_iv: "9cce9dd7751ba549d03306115ee16ae2"
server_key_version: "encryptionkey@0"
I20260929 18:32:45.395042 5707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:45.409097 5707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:45.413838 5707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:45.414079 5707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:45.424535 5707 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:45245
I20260929 18:32:45.424556 5774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:45245 every 8 connection(s)
I20260929 18:32:45.426121 5707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:32:45.430967 5775 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:32:45.432767 5775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462: Bootstrap starting.
I20260929 18:32:45.434063 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 5707
I20260929 18:32:45.434620 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.65:34175
--local_ip_for_outbound_sockets=127.28.207.65
--tserver_master_addrs=127.28.207.126:45245
--webserver_port=32785
--webserver_interface=127.28.207.65
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:35539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.207.65
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:32:45.450228 5775 log.cc:824] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462: Log is configured to *not* fsync() on all Append() calls
I20260929 18:32:45.457252 5775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:32:45.457640 5775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462: Bootstrap complete.
I20260929 18:32:45.462261 5775 raft_consensus.cc:348] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c605752334ab4ed897fd456b3e0e4462" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 45245 } }
I20260929 18:32:45.462708 5775 raft_consensus.cc:729] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c605752334ab4ed897fd456b3e0e4462, State: Initialized, Role: FOLLOWER
I20260929 18:32:45.463259 5775 consensus_queue.cc:260] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c605752334ab4ed897fd456b3e0e4462" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 45245 } }
I20260929 18:32:45.463409 5775 raft_consensus.cc:388] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:32:45.463493 5775 raft_consensus.cc:482] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:32:45.463603 5775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462 [term 1 FOLLOWER]: Advancing to term 2
I20260929 18:32:45.465678 5775 raft_consensus.cc:504] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c605752334ab4ed897fd456b3e0e4462" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 45245 } }
I20260929 18:32:45.466040 5775 leader_election.cc:302] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c605752334ab4ed897fd456b3e0e4462; no voters:
I20260929 18:32:45.466405 5775 leader_election.cc:288] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 18:32:45.466542 5779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 18:32:45.466764 5779 raft_consensus.cc:686] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462 [term 2 LEADER]: Becoming Leader. State: Replica: c605752334ab4ed897fd456b3e0e4462, State: Running, Role: LEADER
I20260929 18:32:45.467136 5779 consensus_queue.cc:237] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c605752334ab4ed897fd456b3e0e4462" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 45245 } }
I20260929 18:32:45.467823 5775 sys_catalog.cc:564] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:32:45.468261 5781 sys_catalog.cc:455] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c605752334ab4ed897fd456b3e0e4462. Latest consensus state: current_term: 2 leader_uuid: "c605752334ab4ed897fd456b3e0e4462" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c605752334ab4ed897fd456b3e0e4462" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 45245 } } }
I20260929 18:32:45.468441 5781 sys_catalog.cc:458] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462 [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:45.468632 5780 sys_catalog.cc:455] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c605752334ab4ed897fd456b3e0e4462" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c605752334ab4ed897fd456b3e0e4462" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 45245 } } }
I20260929 18:32:45.468820 5780 sys_catalog.cc:458] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462 [sys.catalog]: This master's current role is: LEADER
I20260929 18:32:45.468870 5783 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:32:45.472393 5783 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:32:45.473289 5783 catalog_manager.cc:1259] Loaded cluster ID: 09ad7be156324e94bb17e697afda34ff
I20260929 18:32:45.473354 5783 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:32:45.475823 5783 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:32:45.476491 5783 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c605752334ab4ed897fd456b3e0e4462: Loaded TSK: 0
I20260929 18:32:45.477782 5783 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 18:32:45.588784 5777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:45.589216 5777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:45.589298 5777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:45.594053 5777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:45.594190 5777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:45.594302 5777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.65
I20260929 18:32:45.601336 5777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:35539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.65.keytab
--principal=oryx/127.28.207.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.65:34175
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.207.65
--webserver_port=32785
--tserver_master_addrs=127.28.207.126:45245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.65
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:45.602937 5777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:45.604274 5777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:45.617168 5801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:45.617379 5802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:45.617170 5804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:45.617709 5777 server_base.cc:1034] running on GCE node
I20260929 18:32:45.618191 5777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:45.619244 5777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:45.620530 5777 hybrid_clock.cc:648] HybridClock initialized: now 1790706765620520 us; error 50 us; skew 500 ppm
Sep 29 18:32:45 dist-test-slave-qcpv krb5kdc[5194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706765, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.207.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:45.624114 5777 init.cc:377] Logged in from keytab as oryx/127.28.207.65@KRBTEST.COM (short username oryx)
I20260929 18:32:45.625612 5777 webserver.cc:460] Webserver started at http://127.28.207.65:32785/ using document root <none> and password file <none>
I20260929 18:32:45.626300 5777 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:45.626389 5777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:45.631933 5777 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:32:45.635141 5811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:45.637234 5777 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:32:45.637462 5777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-0/wal
uuid: "24d6415b001f4ad9b42c93ca0b0f296e"
format_stamp: "Formatted at 2026-09-29 18:32:44 on dist-test-slave-qcpv"
server_key: "0532ea416869e1e2e5420dd32567e518"
server_key_iv: "926f211cdaa915aca95316893431a058"
server_key_version: "encryptionkey@0"
I20260929 18:32:45.638149 5777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:45.668675 5777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:45.673173 5777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:45.673378 5777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:45.674077 5777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:32:45.675130 5777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:45.675185 5777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:45.675235 5777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:45.675251 5777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:45.688467 5777 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.65:34175
I20260929 18:32:45.688540 5924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.65:34175 every 8 connection(s)
I20260929 18:32:45.689903 5777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
I20260929 18:32:45.698900 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 5777
I20260929 18:32:45.699524 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.66:34743
--local_ip_for_outbound_sockets=127.28.207.66
--tserver_master_addrs=127.28.207.126:45245
--webserver_port=45511
--webserver_interface=127.28.207.66
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:35539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.207.66
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:32:45 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.65@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:45 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.65@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:45.707990 5927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:45.692588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.65:45219)
0929 18:32:45.693246 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:45.693301 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:45.694821 (+ 1520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:45.696464 (+ 1643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:45.696473 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:45.696946 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:45.697480 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:45.697497 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:45.699368 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:45.699374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:45.701004 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:45.701024 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:45.702757 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:45.702760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:45.703017 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:45.703743 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:45.703762 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:45.707112 (+ 3350us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":175,"threads_started":1}
W20260929 18:32:45.708616 5925 heartbeater.cc:644] Failed to heartbeat to 127.28.207.126:45245 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.207.126:45245: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
W20260929 18:32:45.709010 5928 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:45.692630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:45219 (local address 127.28.207.126:45245)
0929 18:32:45.693259 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:45.693281 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:45.693456 (+ 175us) server_negotiation.cc:407] Connection header received
0929 18:32:45.695515 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:45.695529 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:45.695923 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:45.696284 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:45.697710 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:45.699131 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:45.701242 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:45.702401 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:45.702845 (+ 444us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:45.707643 (+ 4798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.65:45219: BlockingRecv error: recv got EOF from 127.28.207.65:45219 (error 108)
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":161,"threads_started":1}
Sep 29 18:32:45 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.65@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:45 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.65@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:45.716724 5927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:45.709169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.65:48013)
0929 18:32:45.709368 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:45.709384 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:45.709478 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:45.710289 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:45.710292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:45.710462 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:45.710592 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:45.710600 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:45.711911 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:45.711914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:45.712542 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:45.712554 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:45.713360 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:45.713363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:45.713447 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:45.714311 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:45.714324 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:45.716578 (+ 2254us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122}
W20260929 18:32:45.716961 5928 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:45.709319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:48013 (local address 127.28.207.126:45245)
0929 18:32:45.709877 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:45.709885 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:45.709909 (+ 24us) server_negotiation.cc:407] Connection header received
0929 18:32:45.709989 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:45.709995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:45.710078 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:45.710209 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:45.710738 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:45.711750 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:45.712719 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:45.713178 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:45.713325 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:45.716819 (+ 3494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.65:48013: BlockingRecv error: recv got EOF from 127.28.207.65:48013 (error 108)
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":176,"threads_started":1}
Sep 29 18:32:45 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.65@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:45 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.65@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:45.726090 5927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:45.717479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.65:40821)
0929 18:32:45.717608 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:45.717625 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:45.717709 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:45.718068 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:45.718071 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:45.718280 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:45.718414 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:45.718423 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:45.719963 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:45.719966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:45.720971 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:45.720982 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:45.721848 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:45.721852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:45.721956 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:45.722659 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:45.722671 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:45.725938 (+ 3267us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 18:32:45.726452 5928 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:45.717546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:40821 (local address 127.28.207.126:45245)
0929 18:32:45.717694 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:45.717699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:45.717716 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:32:45.717815 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:45.717822 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:45.717891 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:45.717981 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:45.718682 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:45.719791 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:45.721177 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:45.721697 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:45.721846 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:45.726274 (+ 4428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.65:40821: BlockingRecv error: recv got EOF from 127.28.207.65:40821 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 18:32:45.726472 5925 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:32:45.855814 5929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:45.856242 5929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:45.856297 5929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:45.861406 5929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:45.861515 5929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:45.861591 5929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.66
I20260929 18:32:45.868201 5929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:35539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.66.keytab
--principal=oryx/127.28.207.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.66:34743
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.207.66
--webserver_port=45511
--tserver_master_addrs=127.28.207.126:45245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.66
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:45.869885 5929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:45.871546 5929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:45.886960 5938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:45.887048 5929 server_base.cc:1034] running on GCE node
W20260929 18:32:45.886981 5936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:45.887415 5935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:45.887964 5929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:45.888693 5929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:45.891996 5929 hybrid_clock.cc:648] HybridClock initialized: now 1790706765891965 us; error 60 us; skew 500 ppm
Sep 29 18:32:45 dist-test-slave-qcpv krb5kdc[5194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706765, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.207.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:45.895763 5929 init.cc:377] Logged in from keytab as oryx/127.28.207.66@KRBTEST.COM (short username oryx)
I20260929 18:32:45.896929 5929 webserver.cc:460] Webserver started at http://127.28.207.66:45511/ using document root <none> and password file <none>
I20260929 18:32:45.897679 5929 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:45.897786 5929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:45.903232 5929 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:32:45.906191 5945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:45.908053 5929 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:32:45.908258 5929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-1/wal
uuid: "812d22c4e03e4d649b68018f12253819"
format_stamp: "Formatted at 2026-09-29 18:32:44 on dist-test-slave-qcpv"
server_key: "3a7819982c98d46add89c81295e97a50"
server_key_iv: "9b78bd37cd9fe4ddde8a9ee83cf3bd28"
server_key_version: "encryptionkey@0"
I20260929 18:32:45.908903 5929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:45.930830 5929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:45.934594 5929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:45.934922 5929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:45.935779 5929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:32:45.936791 5929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:45.936877 5929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:45.936961 5929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:45.937012 5929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:45.948868 5929 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.66:34743
I20260929 18:32:45.948926 6058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.66:34743 every 8 connection(s)
I20260929 18:32:45.950014 5929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
I20260929 18:32:45.950323 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 5929
I20260929 18:32:45.950958 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.67:39471
--local_ip_for_outbound_sockets=127.28.207.67
--tserver_master_addrs=127.28.207.126:45245
--webserver_port=37933
--webserver_interface=127.28.207.67
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:35539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.28.207.67
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:32:45 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.66@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:45 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.66@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 18:32:45.966073 5928 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:45.952382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:42463 (local address 127.28.207.126:45245)
0929 18:32:45.952520 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:45.952524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:45.954079 (+ 1555us) server_negotiation.cc:407] Connection header received
0929 18:32:45.956015 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:45.956021 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:45.956112 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:45.956273 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:45.957795 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:45.958716 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:45.960491 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:45.960949 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:45.961120 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:45.965929 (+ 4809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.66:42463: BlockingRecv error: recv got EOF from 127.28.207.66:42463 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:32:45.966642 6062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:45.953165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.66:42463)
0929 18:32:45.953860 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:45.953926 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:45.955724 (+ 1798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:45.956547 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:45.956558 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:45.957009 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:45.957600 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:45.957619 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:45.959007 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:45.959011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:45.960319 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:45.960335 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:45.961178 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:45.961182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:45.961478 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:45.962307 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:45.962328 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:45.965504 (+ 3176us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":199,"threads_started":1}
W20260929 18:32:45.967717 6059 heartbeater.cc:644] Failed to heartbeat to 127.28.207.126:45245 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.207.126:45245: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Sep 29 18:32:45 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.66@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:45 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.66@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:45.976110 6062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:45.968353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.66:55583)
0929 18:32:45.968571 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:45.968588 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:45.968696 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:45.969235 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:45.969238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:45.969453 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:45.969835 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:45.969852 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:45.971026 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:45.971032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:45.971776 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:45.971787 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:45.972492 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:45.972495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:45.972776 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:45.973746 (+ 970us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:45.973760 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:45.975963 (+ 2203us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128}
W20260929 18:32:45.976341 5928 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:45.968579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:55583 (local address 127.28.207.126:45245)
0929 18:32:45.968870 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:45.968878 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:45.968901 (+ 23us) server_negotiation.cc:407] Connection header received
0929 18:32:45.968968 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:45.968972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:45.969046 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:45.969178 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:45.970007 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:45.970851 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:45.971922 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:45.972280 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:45.972406 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:45.976182 (+ 3776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.66:55583: BlockingRecv error: recv got EOF from 127.28.207.66:55583 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 29 18:32:45 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.66@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:45 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.66@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:45.983793 6062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:45.976932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.66:57737)
0929 18:32:45.977066 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:45.977082 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:45.977170 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:45.977483 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:45.977486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:45.977648 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:45.977761 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:45.977767 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:45.978817 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:45.978821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:45.979660 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:45.979672 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:45.980288 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:45.980292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:45.980367 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:45.980949 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:45.980963 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:45.983593 (+ 2630us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 18:32:45.984203 5928 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:45.977027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:57737 (local address 127.28.207.126:45245)
0929 18:32:45.977163 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:45.977167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:45.977187 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:32:45.977264 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:45.977267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:45.977315 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:45.977407 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:45.977897 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:45.978666 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:45.979799 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:45.980127 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:45.980257 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:45.983803 (+ 3546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.66:57737: BlockingRecv error: recv got EOF from 127.28.207.66:57737 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260929 18:32:45.985178 6059 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:32:46.077072 6061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:32:46.077412 6061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:32:46.077491 6061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:32:46.081655 6061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:32:46.081759 6061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:32:46.081857 6061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.67
I20260929 18:32:46.087329 6061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:35539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/krb5kdc/oryx_127.28.207.67.keytab
--principal=oryx/127.28.207.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.67:39471
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.207.67
--webserver_port=37933
--tserver_master_addrs=127.28.207.126:45245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.67
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:32:46.088519 6061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:32:46.089545 6061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:32:46.100277 6068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:32:46.100299 6070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:46.100474 6061 server_base.cc:1034] running on GCE node
W20260929 18:32:46.100775 6067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:32:46.101221 6061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:32:46.101917 6061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:32:46.103201 6061 hybrid_clock.cc:648] HybridClock initialized: now 1790706766103161 us; error 62 us; skew 500 ppm
Sep 29 18:32:46 dist-test-slave-qcpv krb5kdc[5194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706766, etypes {rep=17 tkt=17 ses=17}, oryx/127.28.207.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:32:46.106479 6061 init.cc:377] Logged in from keytab as oryx/127.28.207.67@KRBTEST.COM (short username oryx)
I20260929 18:32:46.107525 6061 webserver.cc:460] Webserver started at http://127.28.207.67:37933/ using document root <none> and password file <none>
I20260929 18:32:46.108158 6061 fs_manager.cc:359] Metadata directory not provided
I20260929 18:32:46.108242 6061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:32:46.112244 6061 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:32:46.115154 6077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:46.116866 6061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:32:46.117002 6061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-2/wal
uuid: "9c06d48ff10c40aab898a92bbc58226c"
format_stamp: "Formatted at 2026-09-29 18:32:44 on dist-test-slave-qcpv"
server_key: "f6d63fa3293c918ad12ae6196c1b1e11"
server_key_iv: "6a8c6660af76a50e720ffc35c05e1fd1"
server_key_version: "encryptionkey@0"
I20260929 18:32:46.117637 6061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:32:46.131484 6061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:32:46.138149 6061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:32:46.138447 6061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:32:46.139480 6061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:32:46.140761 6061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:32:46.140830 6061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:46.140894 6061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:32:46.140929 6061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:32:46.153738 6061 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.67:39471
I20260929 18:32:46.153761 6190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.67:39471 every 8 connection(s)
I20260929 18:32:46.155139 6061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
I20260929 18:32:46.160480 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 6061
Sep 29 18:32:46 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.67@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:46 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.67@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 18:32:46.169831 5928 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:46.157538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:36175 (local address 127.28.207.126:45245)
0929 18:32:46.157723 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:46.157727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:46.158852 (+ 1125us) server_negotiation.cc:407] Connection header received
0929 18:32:46.160357 (+ 1505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:46.160361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:46.160419 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:46.160522 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:46.161884 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:46.162628 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:46.164301 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:46.164531 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:46.164648 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:46.169674 (+ 5026us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.67:36175: BlockingRecv error: recv got EOF from 127.28.207.67:36175 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 18:32:46.170554 6193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:46.158024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.67:36175)
0929 18:32:46.158663 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:46.158718 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:46.160123 (+ 1405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:46.160738 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:46.160749 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:46.161158 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:46.161695 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:46.161715 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:46.162814 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:46.162819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:46.164167 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:46.164177 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:46.164698 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:46.164702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:46.164872 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:46.165653 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:46.165674 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:46.169342 (+ 3668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":150,"threads_started":1}
W20260929 18:32:46.171278 6191 heartbeater.cc:644] Failed to heartbeat to 127.28.207.126:45245 (0 consecutive failures): Not authorized: Failed to ping master at 127.28.207.126:45245: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
I20260929 18:32:46.173888 5930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:46.161978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60554 (local address 127.28.207.126:45245)
0929 18:32:46.162110 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:46.162123 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:46.162135 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:32:46.162166 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:46.162168 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:46.162205 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:46.162299 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:46.163583 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:46.164270 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:46.165490 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:46.165860 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:46.165997 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:46.168357 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:32:46.168364 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:32:46.168386 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:32:46.168422 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:32:46.171027 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:46.172011 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:46.172018 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:46.172024 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:46.172213 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:32:46.172589 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:32:46.172595 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:32:46.172599 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:32:46.172970 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:32:46.173183 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 18:32:46.173476 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 18:32:46.173704 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 18:32:46 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.67@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:46 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.67@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:46.179272 6193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:46.171976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.67:51609)
0929 18:32:46.172138 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:46.172157 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:46.172264 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:46.172547 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:46.172554 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:46.172746 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:46.172893 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:46.172899 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:46.173888 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:46.173892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:46.175116 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:46.175127 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:46.175630 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:46.175634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:46.175746 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:46.176435 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:46.176450 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:46.179127 (+ 2677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":64}
W20260929 18:32:46.179394 5928 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:46.172070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:51609 (local address 127.28.207.126:45245)
0929 18:32:46.172194 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:46.172198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:46.172214 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:32:46.172389 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:46.172391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:46.172426 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:46.172503 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:46.173007 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:46.173740 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:46.175276 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:46.175494 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:46.175615 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:46.179276 (+ 3661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.67:51609: BlockingRecv error: recv got EOF from 127.28.207.67:51609 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 18:32:46 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.67@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:46 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.67@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:46.185853 6193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:46.180028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.67:37461)
0929 18:32:46.180223 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:46.180254 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:46.180365 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:46.180633 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:46.180636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:46.180813 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:46.180904 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:46.180909 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:46.181673 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:46.181679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:46.182236 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:46.182243 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:46.182682 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:46.182686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:46.182765 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:46.183324 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:46.183336 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:46.185727 (+ 2391us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99}
W20260929 18:32:46.186203 6191 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:32:46.186707 5928 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:46.180166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:37461 (local address 127.28.207.126:45245)
0929 18:32:46.180352 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:46.180357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:46.180372 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:32:46.180492 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:46.180494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:46.180531 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:46.180613 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:46.181047 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:46.181570 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:46.182352 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:46.182571 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:46.182677 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:46.186197 (+ 3520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.67:37461: BlockingRecv error: recv got EOF from 127.28.207.67:37461 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Sep 29 18:32:46 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.65@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:46 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.65@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:46.733925 6195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:46.727125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.65:35595)
0929 18:32:46.727551 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:46.727568 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:46.727734 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:46.728046 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:46.728049 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:46.728278 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:46.728422 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:46.728430 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:46.729254 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:46.729258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:46.729918 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:46.729928 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:46.730473 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:46.730476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:46.730579 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:46.731199 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:46.731212 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:46.733687 (+ 2475us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":111,"threads_started":1}
W20260929 18:32:46.734019 6196 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:46.727324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:35595 (local address 127.28.207.126:45245)
0929 18:32:46.727577 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:46.727580 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:46.727680 (+ 100us) server_negotiation.cc:407] Connection header received
0929 18:32:46.727830 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:46.727832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:46.727882 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:46.727972 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:46.728557 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:46.729116 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:46.730064 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:46.730313 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:46.730427 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:46.733862 (+ 3435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.65:35595: BlockingRecv error: recv got EOF from 127.28.207.65:35595 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":100,"threads_started":1}
Sep 29 18:32:46 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.66@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:46 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.66@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:46.992730 6197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:46.986258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.66:38887)
0929 18:32:46.986637 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:46.986655 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:46.986814 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:46.987264 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:46.987267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:46.987493 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:46.987626 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:46.987634 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:46.988522 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:46.988525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:46.989139 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:46.989148 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:46.989751 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:46.989754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:46.989828 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:46.990530 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:46.990545 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:46.992583 (+ 2038us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":99,"threads_started":1}
W20260929 18:32:46.993059 6196 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:46.986369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:38887 (local address 127.28.207.126:45245)
0929 18:32:46.986609 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:46.986614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:46.986794 (+ 180us) server_negotiation.cc:407] Connection header received
0929 18:32:46.987011 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:46.987018 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:46.987084 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:46.987202 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:46.987780 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:46.988372 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:46.989317 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:46.989593 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:46.989711 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:46.992897 (+ 3186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.66:38887: BlockingRecv error: recv got EOF from 127.28.207.66:38887 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
Sep 29 18:32:47 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.67@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:47 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.67@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:47.193959 6198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:47.187355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.67:53393)
0929 18:32:47.187808 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:47.187830 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:47.188026 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:47.188320 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:47.188323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:47.188575 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:47.188669 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:47.188675 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:47.189494 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:47.189499 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:47.190115 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:47.190123 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:47.190614 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:47.190618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:47.190707 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:47.191345 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:47.191357 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:47.193776 (+ 2419us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":154,"threads_started":1}
W20260929 18:32:47.194316 6196 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:47.187443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:53393 (local address 127.28.207.126:45245)
0929 18:32:47.187678 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:47.187683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:47.187956 (+ 273us) server_negotiation.cc:407] Connection header received
0929 18:32:47.188146 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:47.188149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:47.188205 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:47.188299 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:47.188795 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:47.189371 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:47.190241 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:47.190482 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:47.190606 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:47.194164 (+ 3558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.67:53393: BlockingRecv error: recv got EOF from 127.28.207.67:53393 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 29 18:32:47 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.65@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:47 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.65@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:47.742043 6199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:47.734997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.65:36339)
0929 18:32:47.735372 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:47.735395 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:47.735555 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:47.735924 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:47.735927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:47.736476 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:47.736632 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:47.736640 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:47.737519 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:47.737523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:47.738398 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:47.738408 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:47.738995 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:47.738999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:47.739089 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:47.739841 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:47.739854 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:47.741803 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":110,"threads_started":1}
W20260929 18:32:47.742089 6200 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:47.735189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:36339 (local address 127.28.207.126:45245)
0929 18:32:47.735449 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:47.735456 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:47.735476 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:32:47.735697 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:47.735700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:47.735758 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:47.735856 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:47.736775 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:47.737345 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:47.738570 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:47.738839 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:47.738967 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:47.741958 (+ 2991us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.65:36339: BlockingRecv error: recv got EOF from 127.28.207.65:36339 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":66,"threads_started":1}
Sep 29 18:32:47 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.66@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:47 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.66@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:48.001390 6201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:47.993887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.66:55851)
0929 18:32:47.994437 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:47.994459 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:47.994666 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:47.994998 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:47.995001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:47.995255 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:47.995365 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:47.995372 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:47.996401 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:47.996404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:47.997262 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:47.997281 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:47.997783 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:47.997787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:47.997860 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:47.998544 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:47.998557 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:48.001257 (+ 2700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":133,"threads_started":1}
W20260929 18:32:48.001746 6200 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:47.994107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:55851 (local address 127.28.207.126:45245)
0929 18:32:47.994441 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:47.994449 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:47.994585 (+ 136us) server_negotiation.cc:407] Connection header received
0929 18:32:47.994787 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:47.994794 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:47.994873 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:47.994974 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:47.995486 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:47.996280 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:47.997395 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:47.997662 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:47.997778 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:48.001517 (+ 3739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.66:55851: BlockingRecv error: recv got EOF from 127.28.207.66:55851 (error 108)
Metrics: {"server-negotiator.queue_time_us":192}
Sep 29 18:32:48 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.67@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:48 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.67@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:48.203275 6202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:48.195218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.67:38715)
0929 18:32:48.195567 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:48.195583 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:48.195703 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:48.195981 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:48.195984 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:48.196226 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:48.196315 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:48.196321 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:48.197471 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:48.197474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:48.198308 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:48.198318 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:48.198942 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:48.198945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:48.199051 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:48.199809 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:48.199851 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:48.203000 (+ 3149us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":112,"threads_started":1}
W20260929 18:32:48.203369 6200 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:48.195464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:38715 (local address 127.28.207.126:45245)
0929 18:32:48.195617 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:48.195624 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:48.195642 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:32:48.195818 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:48.195821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:48.195874 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:48.195964 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:48.196426 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:48.197254 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:48.198483 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:48.198786 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:48.198936 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:48.203219 (+ 4283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.67:38715: BlockingRecv error: recv got EOF from 127.28.207.67:38715 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 18:32:48 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.65@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:48 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.65@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:48.750030 6203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:48.743204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.65:46635)
0929 18:32:48.743495 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:48.743509 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:48.743610 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:48.744095 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:48.744098 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:48.744312 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:48.744428 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:48.744433 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:48.745457 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:48.745459 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:48.746079 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:48.746088 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:48.746691 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:48.746694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:48.746767 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:48.747407 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:48.747419 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:48.749903 (+ 2484us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":106,"threads_started":1}
W20260929 18:32:48.750327 6204 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:48.743394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:46635 (local address 127.28.207.126:45245)
0929 18:32:48.743691 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:48.743695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:48.743713 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:32:48.743768 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:48.743770 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:48.743823 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:48.744027 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:48.744603 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:48.745209 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:48.746236 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:48.746559 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:48.746666 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:48.750177 (+ 3511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.65:46635: BlockingRecv error: recv got EOF from 127.28.207.65:46635 (error 108)
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":110,"threads_started":1}
Sep 29 18:32:49 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.66@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:49 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.66@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:49.009153 6205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:49.002433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.66:40217)
0929 18:32:49.002718 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:49.002732 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:49.002846 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:49.003196 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:49.003199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:49.003394 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:49.003495 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:49.003501 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:49.004425 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:49.004430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:49.005089 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:49.005101 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:49.005745 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:49.005748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:49.005824 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:49.006519 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:49.006532 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:49.008930 (+ 2398us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":73,"threads_started":1}
W20260929 18:32:49.009168 6204 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:49.002593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:40217 (local address 127.28.207.126:45245)
0929 18:32:49.002777 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:49.002781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:49.002798 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:32:49.003001 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:49.003004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:49.003055 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:49.003147 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:49.003642 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:49.004245 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:49.005363 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:49.005599 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:49.005706 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:49.009067 (+ 3361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.66:40217: BlockingRecv error: recv got EOF from 127.28.207.66:40217 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 29 18:32:49 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.67@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:49 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.67@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:49.216826 6206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:49.204594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.67:60497)
0929 18:32:49.204995 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:49.205016 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:49.205203 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:49.205508 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:49.205511 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:49.205726 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:49.205816 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:49.205821 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:49.206628 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:49.206634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:49.207271 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:49.207279 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:49.207740 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:49.207744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:49.207823 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:49.214215 (+ 6392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:49.214230 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:49.216657 (+ 2427us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":109,"threads_started":1}
W20260929 18:32:49.217096 6204 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:49.204779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:60497 (local address 127.28.207.126:45245)
0929 18:32:49.204994 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:49.205000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:49.205133 (+ 133us) server_negotiation.cc:407] Connection header received
0929 18:32:49.205330 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:49.205333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:49.205389 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:49.205472 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:49.205933 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:49.206517 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:49.207386 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:49.207625 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:49.207731 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:49.216938 (+ 9207us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.67:60497: BlockingRecv error: recv got EOF from 127.28.207.67:60497 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 29 18:32:49 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.65@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:49 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.65@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:49.758193 6207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:49.751343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.65:58499)
0929 18:32:49.751681 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:49.751697 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:49.751864 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:49.752295 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:49.752298 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:49.752586 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:49.752745 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:49.752752 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:49.753655 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:49.753659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:49.754301 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:49.754311 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:49.754971 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:49.754975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:49.755050 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:49.755733 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:49.755745 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:49.758034 (+ 2289us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":120,"threads_started":1}
W20260929 18:32:49.758388 6208 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:49.751440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:58499 (local address 127.28.207.126:45245)
0929 18:32:49.751721 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:49.751726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:49.751813 (+ 87us) server_negotiation.cc:407] Connection header received
0929 18:32:49.752004 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:49.752010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:49.752074 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:49.752178 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:49.752871 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:49.753397 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:49.754486 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:49.754750 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:49.754882 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:49.758228 (+ 3346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.65:58499: BlockingRecv error: recv got EOF from 127.28.207.65:58499 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":109,"threads_started":1}
Sep 29 18:32:50 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.66@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:50 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.66@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:50.017849 6209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:50.010426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.66:34847)
0929 18:32:50.010899 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:50.010920 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:50.011113 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:50.011423 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:50.011426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:50.011695 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:50.011851 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:50.011859 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:50.013086 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:50.013089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:50.013865 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:50.013900 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:50.014414 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:50.014418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:50.014495 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:50.015168 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:50.015181 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:50.017688 (+ 2507us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":121,"threads_started":1}
W20260929 18:32:50.018126 6208 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:50.010614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:34847 (local address 127.28.207.126:45245)
0929 18:32:50.010899 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:50.010907 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:50.011034 (+ 127us) server_negotiation.cc:407] Connection header received
0929 18:32:50.011237 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:50.011247 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:50.011301 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:50.011405 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:50.012057 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:50.012924 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:50.014033 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:50.014287 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:50.014410 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:50.017963 (+ 3553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.66:34847: BlockingRecv error: recv got EOF from 127.28.207.66:34847 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
Sep 29 18:32:50 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.67@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:50 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.67@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:50.225430 6210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:50.217822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.67:35847)
0929 18:32:50.218123 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:50.218138 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:50.218228 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:50.218526 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:50.218528 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:50.218738 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:50.218843 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:50.218848 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:50.220196 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:50.220200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:50.220872 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:50.220881 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:50.221668 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:50.221671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:50.221741 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:50.222475 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:50.222490 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:50.225214 (+ 2724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":127,"threads_started":1}
W20260929 18:32:50.225597 6208 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:50.218030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:35847 (local address 127.28.207.126:45245)
0929 18:32:50.218197 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:50.218201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:50.218224 (+ 23us) server_negotiation.cc:407] Connection header received
0929 18:32:50.218318 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:50.218321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:50.218378 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:50.218486 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:50.219127 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:50.220048 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:50.220999 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:50.221509 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:50.221611 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:50.225377 (+ 3766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.67:35847: BlockingRecv error: recv got EOF from 127.28.207.67:35847 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 29 18:32:50 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.65@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:50 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.65@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:50.766121 6211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:50.759188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.65:60723)
0929 18:32:50.759604 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:50.759625 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:50.759806 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:50.760159 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:50.760163 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:50.760407 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:50.760670 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:50.760680 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:50.761536 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:50.761539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:50.762189 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:50.762199 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:50.762725 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:50.762728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:50.762806 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:50.763597 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:50.763618 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:50.765942 (+ 2324us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":117,"threads_started":1}
W20260929 18:32:50.766294 6212 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:50.759383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:60723 (local address 127.28.207.126:45245)
0929 18:32:50.759648 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:50.759651 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:50.759751 (+ 100us) server_negotiation.cc:407] Connection header received
0929 18:32:50.759907 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:50.759909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:50.759952 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:50.760054 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:50.760826 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:50.761384 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:50.762335 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:50.762564 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:50.762652 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:50.766133 (+ 3481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.65:60723: BlockingRecv error: recv got EOF from 127.28.207.65:60723 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":106,"threads_started":1}
Sep 29 18:32:51 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.66@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:51 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.66@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:51.033075 6213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:51.019367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.66:44479)
0929 18:32:51.019717 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:51.019739 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:51.019855 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:51.020244 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:51.020246 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:51.020495 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:51.020628 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:51.020634 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:51.021868 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:51.021872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:51.022682 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:51.022700 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:51.023585 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:51.023589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:51.023694 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:51.030408 (+ 6714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:51.030425 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:51.032903 (+ 2478us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":136,"threads_started":1}
W20260929 18:32:51.033537 6212 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:51.019623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:44479 (local address 127.28.207.126:45245)
0929 18:32:51.019785 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:51.019789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:51.019807 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:32:51.020016 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:51.020020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:51.020077 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:51.020187 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:51.020803 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:51.021506 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:51.022847 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:51.023328 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:51.023466 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:51.033398 (+ 9932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.66:44479: BlockingRecv error: recv got EOF from 127.28.207.66:44479 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 18:32:51 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.67@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:51 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.67@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:51.234886 6214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:51.226634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.67:44283)
0929 18:32:51.227393 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:51.227411 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:51.227579 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:51.227925 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:51.227931 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:51.228245 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:51.228433 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:51.228443 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:51.229260 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:51.229265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:51.230439 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:51.230452 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:51.231122 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:51.231127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:51.231246 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:51.231950 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:51.231972 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:51.234658 (+ 2686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":124,"threads_started":1}
W20260929 18:32:51.234923 6212 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:51.226720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:44283 (local address 127.28.207.126:45245)
0929 18:32:51.227185 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:51.227192 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:51.227538 (+ 346us) server_negotiation.cc:407] Connection header received
0929 18:32:51.227712 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:51.227714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:51.227772 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:51.227884 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:51.228567 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:51.229127 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:51.230570 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:51.230808 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:51.230948 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:51.234794 (+ 3846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.67:44283: BlockingRecv error: recv got EOF from 127.28.207.67:44283 (error 108)
Metrics: {"server-negotiator.queue_time_us":311}
Sep 29 18:32:51 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.65@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:51 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.65@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:51.774740 6215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:51.767297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.65:43325)
0929 18:32:51.767705 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:51.767731 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:51.767950 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:51.768686 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:51.768689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:51.768904 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:51.769092 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:51.769100 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:51.770304 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:51.770307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:51.771098 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:51.771109 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:51.772044 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:51.772047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:51.772117 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:51.772741 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:51.772753 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:51.774604 (+ 1851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":115,"threads_started":1}
W20260929 18:32:51.775251 6216 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:51.767568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:43325 (local address 127.28.207.126:45245)
0929 18:32:51.768005 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:51.768010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:51.768030 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:32:51.768415 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:51.768419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:51.768498 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:51.768624 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:51.769247 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:51.770152 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:51.771271 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:51.771649 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:51.771788 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:51.774993 (+ 3205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.65:43325: BlockingRecv error: recv got EOF from 127.28.207.65:43325 (error 108)
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":116,"threads_started":1}
Sep 29 18:32:52 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.66@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:52 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.66@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:52.041849 6217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:52.034119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.66:53887)
0929 18:32:52.034627 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:52.034643 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:52.034832 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:52.035201 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:52.035204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:52.035478 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:52.035587 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:52.035593 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:52.036874 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:52.036878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:52.037826 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:52.037837 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:52.038383 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:52.038386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:52.038466 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:52.039385 (+ 919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:52.039403 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:52.041710 (+ 2307us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":152,"threads_started":1}
W20260929 18:32:52.042315 6216 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:52.034194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:53887 (local address 127.28.207.126:45245)
0929 18:32:52.034497 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:52.034503 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:52.034755 (+ 252us) server_negotiation.cc:407] Connection header received
0929 18:32:52.034990 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:52.034993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:52.035056 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:52.035185 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:52.035714 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:52.036709 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:52.037975 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:52.038248 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:52.038372 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:52.042164 (+ 3792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.66:53887: BlockingRecv error: recv got EOF from 127.28.207.66:53887 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 29 18:32:52 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.67@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:52 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.67@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:52.243844 6218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:52.236228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.67:50103)
0929 18:32:52.236508 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:52.236529 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:52.236680 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:52.237248 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:52.237252 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:52.237488 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:52.237645 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:52.237656 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:52.238742 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:52.238746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:52.239544 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:52.239553 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:52.240130 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:52.240132 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:52.240200 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:52.240899 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:52.240911 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:52.243688 (+ 2777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":110,"threads_started":1}
W20260929 18:32:52.244094 6216 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:52.236444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:50103 (local address 127.28.207.126:45245)
0929 18:32:52.236626 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:52.236630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:52.236652 (+ 22us) server_negotiation.cc:407] Connection header received
0929 18:32:52.236926 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:52.236931 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:52.237012 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:52.237146 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:52.237824 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:52.238565 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:52.239735 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:52.239996 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:52.240099 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:52.243922 (+ 3823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.67:50103: BlockingRecv error: recv got EOF from 127.28.207.67:50103 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 29 18:32:52 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.65@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:52 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.65@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:52.782932 6220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:52.775681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.65:34777)
0929 18:32:52.776062 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:52.776076 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:52.776169 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:52.776589 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:52.776592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:52.776787 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:52.776891 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:52.776897 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:52.777883 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:52.777886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:52.778629 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:52.778644 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:52.779625 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:52.779628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:52.779695 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:52.780424 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:52.780436 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:52.782730 (+ 2294us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":106,"threads_started":1}
W20260929 18:32:52.783233 6219 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:52.775716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:34777 (local address 127.28.207.126:45245)
0929 18:32:52.776097 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:52.776101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:52.776130 (+ 29us) server_negotiation.cc:407] Connection header received
0929 18:32:52.776349 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:52.776352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:52.776417 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:52.776527 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:52.777055 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:52.777751 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:52.778777 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:52.779483 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:52.779602 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:52.783047 (+ 3445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.65:34777: BlockingRecv error: recv got EOF from 127.28.207.65:34777 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":71,"threads_started":1}
Sep 29 18:32:53 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.66@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:53 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.66@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:53.049786 6221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:53.042840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.66:35221)
0929 18:32:53.043374 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:53.043389 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:53.043562 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:53.043855 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:53.043858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:53.044125 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:53.044237 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:53.044243 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:53.045143 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:53.045148 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:53.045777 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:53.045786 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:53.046293 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:53.046296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:53.046374 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:53.047055 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:53.047076 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:53.049549 (+ 2473us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":162,"threads_started":1}
W20260929 18:32:53.049844 6219 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:53.043064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:35221 (local address 127.28.207.126:45245)
0929 18:32:53.043254 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:53.043259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:53.043484 (+ 225us) server_negotiation.cc:407] Connection header received
0929 18:32:53.043676 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:53.043679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:53.043738 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:53.043835 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:53.044370 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:53.044957 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:53.045921 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:53.046170 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:53.046288 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:53.049721 (+ 3433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.66:35221: BlockingRecv error: recv got EOF from 127.28.207.66:35221 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 29 18:32:53 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.67@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:53 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.67@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:53.251596 6222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:53.244827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.67:55251)
0929 18:32:53.245309 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:53.245325 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:53.245497 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:53.245836 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:53.245838 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:53.246061 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:53.246182 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:53.246188 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:53.247011 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:53.247015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:53.247623 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:53.247633 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:53.248184 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:53.248188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:53.248284 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:53.248930 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:53.248945 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:53.251432 (+ 2487us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":95,"threads_started":1}
W20260929 18:32:53.251812 6219 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:53.245032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:55251 (local address 127.28.207.126:45245)
0929 18:32:53.245195 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:53.245201 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:53.245450 (+ 249us) server_negotiation.cc:407] Connection header received
0929 18:32:53.245622 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:53.245625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:53.245695 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:53.245777 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:53.246293 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:53.246845 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:53.247759 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:53.248008 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:53.248099 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:53.251615 (+ 3516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.67:55251: BlockingRecv error: recv got EOF from 127.28.207.67:55251 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 18:32:53 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.65@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:53 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.65@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:53.792073 6223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:53.784208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.65:38587)
0929 18:32:53.784580 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:53.784596 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:53.784718 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:53.785222 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:53.785225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:53.785450 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:53.785580 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:53.785586 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:53.786480 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:53.786484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:53.787265 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:53.787280 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:53.787840 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:53.787844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:53.787939 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:53.788677 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:53.788694 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:53.791937 (+ 3243us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":144,"threads_started":1}
W20260929 18:32:53.792436 6224 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:53.784415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:38587 (local address 127.28.207.126:45245)
0929 18:32:53.784679 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:53.784683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:53.784699 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:32:53.784896 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:53.784899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:53.784953 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:53.785085 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:53.785766 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:53.786317 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:53.787437 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:53.787666 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:53.787774 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:53.792248 (+ 4474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.65:38587: BlockingRecv error: recv got EOF from 127.28.207.65:38587 (error 108)
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":99,"threads_started":1}
Sep 29 18:32:54 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.66@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:54 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.66@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:54.060807 6225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:54.051114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.66:36051)
0929 18:32:54.051589 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:54.051614 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:54.051910 (+ 296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:54.052419 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:54.052424 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:54.052694 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:54.052842 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:54.052850 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:54.054218 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:54.054222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:54.055195 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:54.055209 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:54.056210 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:54.056216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:54.056341 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:54.057136 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:54.057149 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:54.060604 (+ 3455us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":93,"threads_started":1}
W20260929 18:32:54.061087 6224 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:54.051394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:36051 (local address 127.28.207.126:45245)
0929 18:32:54.051776 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:54.051780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:54.051808 (+ 28us) server_negotiation.cc:407] Connection header received
0929 18:32:54.052088 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:54.052091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:54.052152 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:54.052271 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:54.053123 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:54.054004 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:54.055480 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:54.056008 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:54.056189 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:54.060916 (+ 4727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.66:36051: BlockingRecv error: recv got EOF from 127.28.207.66:36051 (error 108)
Metrics: {"server-negotiator.queue_time_us":198}
Sep 29 18:32:54 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.67@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:54 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.67@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:54.268013 6226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:54.252585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.67:39619)
0929 18:32:54.253064 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:54.253111 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:54.253332 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:54.253663 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:54.253667 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:54.253932 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:54.254046 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:54.254052 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:54.255050 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:54.255054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:54.255832 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:54.255842 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:54.256611 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:54.256619 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:54.256785 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:54.264310 (+ 7525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:54.264327 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:54.267479 (+ 3152us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":117,"threads_started":1}
W20260929 18:32:54.268272 6224 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:54.252777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:39619 (local address 127.28.207.126:45245)
0929 18:32:54.253065 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:54.253072 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:54.253241 (+ 169us) server_negotiation.cc:407] Connection header received
0929 18:32:54.253457 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:54.253461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:54.253526 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:54.253643 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:54.254195 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:54.254801 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:54.255972 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:54.256343 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:54.256478 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:54.267848 (+ 11370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.67:39619: BlockingRecv error: recv got EOF from 127.28.207.67:39619 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
Sep 29 18:32:54 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.65@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:54 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.65@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:54.799873 6227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:54.793059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.65:57569)
0929 18:32:54.793491 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:54.793509 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:54.793665 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:54.794054 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:54.794057 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:54.794295 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:54.794444 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:54.794450 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:54.795404 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:54.795407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:54.796118 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:54.796128 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:54.796687 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:54.796690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:54.796770 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:54.797490 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:54.797502 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:54.799705 (+ 2203us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":135,"threads_started":1}
W20260929 18:32:54.800298 6228 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:54.793307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:57569 (local address 127.28.207.126:45245)
0929 18:32:54.793565 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:54.793568 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:54.793599 (+ 31us) server_negotiation.cc:407] Connection header received
0929 18:32:54.793800 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:54.793803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:54.793853 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:54.793959 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:54.794584 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:54.795272 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:54.796269 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:54.796503 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:54.796610 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:54.799892 (+ 3282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.65:57569: BlockingRecv error: recv got EOF from 127.28.207.65:57569 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":90,"threads_started":1}
Sep 29 18:32:55 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.66@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:55 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.66@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:55.069525 6229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:55.062326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.66:33341)
0929 18:32:55.062736 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:55.062759 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:55.062954 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:55.063321 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:55.063324 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:55.063591 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:55.063702 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:55.063708 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:55.064561 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:55.064567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:55.065152 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:55.065160 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:55.065646 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:55.065649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:55.065726 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:55.066420 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:55.066432 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:55.069385 (+ 2953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":149,"threads_started":1}
W20260929 18:32:55.069811 6228 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:55.062398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:33341 (local address 127.28.207.126:45245)
0929 18:32:55.062562 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:55.062568 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:55.062907 (+ 339us) server_negotiation.cc:407] Connection header received
0929 18:32:55.063132 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:55.063134 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:55.063193 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:55.063288 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:55.063845 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:55.064440 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:55.065279 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:55.065524 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:55.065642 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:55.069681 (+ 4039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.66:33341: BlockingRecv error: recv got EOF from 127.28.207.66:33341 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 29 18:32:55 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.67@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:55 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.67@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:55.276618 6230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:55.269294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.67:56111)
0929 18:32:55.269642 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:55.269662 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:55.269817 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:55.270184 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:55.270187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:55.270430 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:55.270641 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:55.270653 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:55.271686 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:55.271691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:55.272299 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:55.272310 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:55.272824 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:55.272827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:55.272908 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:55.273943 (+ 1035us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:55.273955 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:55.276395 (+ 2440us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":116,"threads_started":1}
W20260929 18:32:55.276703 6228 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:55.269432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:56111 (local address 127.28.207.126:45245)
0929 18:32:55.269659 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:55.269665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:55.269838 (+ 173us) server_negotiation.cc:407] Connection header received
0929 18:32:55.269904 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:55.269906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:55.269976 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:55.270092 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:55.270918 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:55.271499 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:55.272442 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:55.272683 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:55.272824 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:55.276572 (+ 3748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.67:56111: BlockingRecv error: recv got EOF from 127.28.207.67:56111 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 29 18:32:55 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.65@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:55 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.65@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:55.808414 6232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:55.800893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.65:45755)
0929 18:32:55.801476 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:55.801489 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:55.801614 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:55.801906 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:55.801909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:55.802137 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:55.802231 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:55.802238 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:55.803103 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:55.803109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:55.803734 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:55.803743 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:55.804237 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:55.804241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:55.804337 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:55.804970 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:55.804993 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:55.808243 (+ 3250us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":140,"threads_started":1}
W20260929 18:32:55.808573 6231 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:55.800945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:45755 (local address 127.28.207.126:45245)
0929 18:32:55.801240 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:55.801244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:55.801568 (+ 324us) server_negotiation.cc:407] Connection header received
0929 18:32:55.801727 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:55.801730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:55.801784 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:55.801887 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:55.802358 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:55.802993 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:55.803884 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:55.804120 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:55.804231 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:55.808406 (+ 4175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.65:45755: BlockingRecv error: recv got EOF from 127.28.207.65:45755 (error 108)
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":128,"threads_started":1}
Sep 29 18:32:56 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.66@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:56 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.66@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:56.078114 6233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:56.071274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.66:42737)
0929 18:32:56.071641 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:56.071657 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:56.071805 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:56.072196 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:56.072198 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:56.072392 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:56.072499 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:56.072504 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:56.073587 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:56.073590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:56.074204 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:56.074211 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:56.074932 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:56.074936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:56.075008 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:56.075654 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:56.075666 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:56.077988 (+ 2322us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":144,"threads_started":1}
W20260929 18:32:56.078568 6231 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:56.071363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:42737 (local address 127.28.207.126:45245)
0929 18:32:56.071569 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:56.071576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:56.071748 (+ 172us) server_negotiation.cc:407] Connection header received
0929 18:32:56.071976 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:56.071979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:56.072048 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:56.072141 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:56.072661 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:56.073458 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:56.074358 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:56.074612 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:56.074977 (+ 365us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:56.078298 (+ 3321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.66:42737: BlockingRecv error: recv got EOF from 127.28.207.66:42737 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 29 18:32:56 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.67@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:56 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.67@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:56.284720 6234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:56.277799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.67:47565)
0929 18:32:56.278219 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:56.278240 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:56.278394 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:56.278681 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:56.278684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:56.278997 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:56.279090 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:56.279096 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:56.279915 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:56.279922 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:56.280518 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:56.280526 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:56.280994 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:56.280997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:56.281068 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:56.281749 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:56.281760 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:56.284539 (+ 2779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":147,"threads_started":1}
W20260929 18:32:56.284802 6231 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:56.278053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:47565 (local address 127.28.207.126:45245)
0929 18:32:56.278236 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:56.278243 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:56.278334 (+ 91us) server_negotiation.cc:407] Connection header received
0929 18:32:56.278502 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:56.278506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:56.278566 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:56.278665 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:56.279211 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:56.279797 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:56.280641 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:56.280877 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:56.280990 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:56.284679 (+ 3689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.67:47565: BlockingRecv error: recv got EOF from 127.28.207.67:47565 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 29 18:32:56 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.65@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:56 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.65@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:56.816322 6235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:56.809456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.65:50355)
0929 18:32:56.809747 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:56.809761 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:56.809863 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:56.810167 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:56.810170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:56.810349 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:56.810439 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:56.810444 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:56.811323 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:56.811326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:56.811877 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:56.811885 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:56.812641 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:56.812650 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:56.812777 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:56.813603 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:56.813624 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:56.816197 (+ 2573us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":112,"threads_started":1}
W20260929 18:32:56.816715 6236 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:56.809635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:50355 (local address 127.28.207.126:45245)
0929 18:32:56.809849 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:56.809852 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:56.809866 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:32:56.809961 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:56.809963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:56.810007 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:56.810104 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:56.810562 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:56.811188 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:56.812084 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:56.812447 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:56.812561 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:56.816518 (+ 3957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.65:50355: BlockingRecv error: recv got EOF from 127.28.207.65:50355 (error 108)
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":60,"threads_started":1}
Sep 29 18:32:57 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.66@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:57 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.66@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:57.086004 6237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:57.079338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.66:40857)
0929 18:32:57.079781 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:57.079801 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:57.079987 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:57.080322 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:57.080325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:57.080595 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:57.080729 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:57.080735 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:57.081570 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:57.081575 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:57.082162 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:57.082169 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:57.082649 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:57.082653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:57.082736 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:57.083485 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:57.083504 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:57.085838 (+ 2334us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":97,"threads_started":1}
W20260929 18:32:57.086476 6236 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:57.079500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:40857 (local address 127.28.207.126:45245)
0929 18:32:57.079747 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:57.079753 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:57.079905 (+ 152us) server_negotiation.cc:407] Connection header received
0929 18:32:57.080117 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:57.080122 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:57.080198 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:57.080296 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:57.080854 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:57.081455 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:57.082287 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:57.082528 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:57.082644 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:57.086321 (+ 3677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.66:40857: BlockingRecv error: recv got EOF from 127.28.207.66:40857 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 29 18:32:57 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.67@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:57 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.67@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:57.293119 6238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:57.285875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.67:46987)
0929 18:32:57.286414 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:57.286429 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:57.286551 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:57.286993 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:57.286995 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:57.287186 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:57.287320 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:57.287377 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:57.288407 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:57.288411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:57.289009 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:57.289017 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:57.289620 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:57.289623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:57.289700 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:57.290376 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:57.290390 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:57.292861 (+ 2471us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":125,"threads_started":1}
W20260929 18:32:57.293341 6236 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:57.285965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:46987 (local address 127.28.207.126:45245)
0929 18:32:57.286205 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:57.286211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:57.286535 (+ 324us) server_negotiation.cc:407] Connection header received
0929 18:32:57.286665 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:57.286668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:57.286744 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:57.286842 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:57.287546 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:57.288160 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:57.289192 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:57.289485 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:57.289587 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:57.293127 (+ 3540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.67:46987: BlockingRecv error: recv got EOF from 127.28.207.67:46987 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 29 18:32:57 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.65@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:57 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.65@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:57.824060 6239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:57.817311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.65:41691)
0929 18:32:57.817708 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:57.817726 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:57.817886 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:57.818264 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:57.818267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:57.818497 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:57.818623 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:57.818629 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:57.819525 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:57.819529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:57.820234 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:57.820245 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:57.821005 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:57.821016 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:57.821148 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:57.821868 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:57.821887 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:57.823937 (+ 2050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":106,"threads_started":1}
W20260929 18:32:57.824448 6240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:57.817536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:41691 (local address 127.28.207.126:45245)
0929 18:32:57.817833 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:57.817836 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:57.817854 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:32:57.817998 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:57.818001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:57.818053 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:57.818175 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:57.818779 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:57.819357 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:57.820436 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:57.820761 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:57.820961 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:57.824285 (+ 3324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.65:41691: BlockingRecv error: recv got EOF from 127.28.207.65:41691 (error 108)
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":112,"threads_started":1}
Sep 29 18:32:58 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.66@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:58 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.66@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:58.094580 6241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:58.087107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.66:60687)
0929 18:32:58.087507 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:58.087526 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:58.087682 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:58.087973 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:58.087976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:58.088216 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:58.088310 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:58.088316 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:58.089615 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:58.089620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:58.090522 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:58.090534 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:58.091314 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:58.091317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:58.091404 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:58.092012 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:58.092028 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:58.094428 (+ 2400us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":124,"threads_started":1}
W20260929 18:32:58.094808 6240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:58.087324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:60687 (local address 127.28.207.126:45245)
0929 18:32:58.087513 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:58.087521 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:58.087620 (+ 99us) server_negotiation.cc:407] Connection header received
0929 18:32:58.087806 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:58.087810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:58.087865 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:58.087957 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:58.088439 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:58.089423 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:58.090730 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:58.091147 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:58.091256 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:58.094637 (+ 3381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.66:60687: BlockingRecv error: recv got EOF from 127.28.207.66:60687 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 29 18:32:58 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.67@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:58 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.67@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:58.301604 6242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:58.294245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.67:33597)
0929 18:32:58.294587 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:58.294612 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:58.294752 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:58.295228 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:58.295242 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:58.295719 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:58.295888 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:58.295897 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:58.296870 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:58.296873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:58.297511 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:58.297520 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:58.298092 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:58.298095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:58.298165 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:58.298801 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:58.298815 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:58.301429 (+ 2614us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":114,"threads_started":1}
W20260929 18:32:58.301821 6240 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:58.294369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:33597 (local address 127.28.207.126:45245)
0929 18:32:58.294559 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:58.294563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:58.294719 (+ 156us) server_negotiation.cc:407] Connection header received
0929 18:32:58.294909 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:58.294921 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:58.294989 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:58.295095 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:58.296036 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:58.296744 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:58.297660 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:58.297928 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:58.298049 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:58.301634 (+ 3585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.67:33597: BlockingRecv error: recv got EOF from 127.28.207.67:33597 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 18:32:58 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.65@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:58 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.65@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:58.832729 6243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:58.825077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.65:35069)
0929 18:32:58.825483 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:58.825502 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:58.825664 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:58.826077 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:58.826081 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:58.826313 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:58.826440 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:58.826446 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:58.827674 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:58.827679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:58.828547 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:58.828560 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:58.829248 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:58.829251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:58.829326 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:58.830026 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:58.830039 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:58.832521 (+ 2482us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":112,"threads_started":1}
W20260929 18:32:58.832844 6244 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:58.825298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:35069 (local address 127.28.207.126:45245)
0929 18:32:58.825569 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:58.825573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:58.825593 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:32:58.825796 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:58.825799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:58.825859 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:58.825962 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:58.826608 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:58.827456 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:58.828719 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:58.829094 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:58.829195 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:58.832668 (+ 3473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.65:35069: BlockingRecv error: recv got EOF from 127.28.207.65:35069 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":97,"threads_started":1}
Sep 29 18:32:59 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.66@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:59 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.66@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:59.104125 6245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:59.095855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.66:36385)
0929 18:32:59.096363 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:59.096383 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:59.096574 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:59.097168 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:59.097175 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:59.097432 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:59.097561 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:59.097569 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:59.098540 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:59.098543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:59.099315 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:59.099327 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:59.099896 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:59.099900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:59.099971 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:59.100926 (+ 955us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:59.100940 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:59.103945 (+ 3005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":109,"threads_started":1}
W20260929 18:32:59.104641 6244 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:59.096088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:36385 (local address 127.28.207.126:45245)
0929 18:32:59.096365 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:59.096372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:59.096498 (+ 126us) server_negotiation.cc:407] Connection header received
0929 18:32:59.096791 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:59.096798 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:59.096876 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:59.096972 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:59.097780 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:59.098332 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:59.099455 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:59.099733 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:59.099851 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:59.104401 (+ 4550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.66:36385: BlockingRecv error: recv got EOF from 127.28.207.66:36385 (error 108)
Metrics: {"server-negotiator.queue_time_us":154}
Sep 29 18:32:59 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.67@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:59 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.67@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:59.319352 6246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:59.302663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.67:50001)
0929 18:32:59.303085 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:59.303117 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:59.303271 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:59.303550 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:59.303553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:59.303781 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:59.303877 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:59.303882 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:59.304653 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:59.304659 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:59.305282 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:59.305290 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:59.305760 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:59.305764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:59.305844 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:59.316230 (+ 10386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:59.316245 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:59.319016 (+ 2771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":116,"threads_started":1}
W20260929 18:32:59.319389 6244 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:59.302825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:50001 (local address 127.28.207.126:45245)
0929 18:32:59.303023 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:59.303029 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:59.303203 (+ 174us) server_negotiation.cc:407] Connection header received
0929 18:32:59.303385 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:59.303388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:59.303444 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:59.303527 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:59.303994 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:59.304527 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:59.305400 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:59.305642 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:59.305751 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:59.319226 (+ 13475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.67:50001: BlockingRecv error: recv got EOF from 127.28.207.67:50001 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 29 18:32:59 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.65@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:32:59 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.65@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:32:59.841087 6247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:32:59.834131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.65:42169)
0929 18:32:59.834729 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0929 18:32:59.834747 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:32:59.834934 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:32:59.835343 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:32:59.835348 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:32:59.835584 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:32:59.835693 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:59.835699 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:59.836612 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:59.836619 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:59.837251 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:32:59.837260 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:32:59.837776 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:32:59.837781 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:32:59.837868 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:59.838520 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:32:59.838533 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:32:59.840926 (+ 2393us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":121,"threads_started":1}
W20260929 18:32:59.841338 6248 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:32:59.834269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:42169 (local address 127.28.207.126:45245)
0929 18:32:59.834625 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0929 18:32:59.834629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:32:59.834881 (+ 252us) server_negotiation.cc:407] Connection header received
0929 18:32:59.835074 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:32:59.835081 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:32:59.835155 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:32:59.835288 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:32:59.835824 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:59.836474 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:59.837382 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:32:59.837649 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:32:59.837773 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:32:59.841123 (+ 3350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.65:42169: BlockingRecv error: recv got EOF from 127.28.207.65:42169 (error 108)
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":96,"threads_started":1}
Sep 29 18:33:00 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.66@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:33:00 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.66@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:33:00.112692 6249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:00.105364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.66:41507)
0929 18:33:00.105977 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:00.106000 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:00.106308 (+ 308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:00.106754 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:00.106757 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:00.107278 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:33:00.107411 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:00.107418 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:00.108544 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:00.108548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:00.109166 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:00.109175 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:00.109660 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:00.109664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:00.109737 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:00.110482 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:33:00.110495 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:33:00.112487 (+ 1992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":177,"threads_started":1}
W20260929 18:33:00.112795 6248 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:33:00.105627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:41507 (local address 127.28.207.126:45245)
0929 18:33:00.105950 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:00.105956 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:00.106229 (+ 273us) server_negotiation.cc:407] Connection header received
0929 18:33:00.106439 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:00.106444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:00.106507 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:00.106736 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:00.107574 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:00.108401 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:00.109296 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:00.109536 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:00.109654 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:00.112642 (+ 2988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.66:41507: BlockingRecv error: recv got EOF from 127.28.207.66:41507 (error 108)
Metrics: {"server-negotiator.queue_time_us":186}
Sep 29 18:33:00 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.67@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:33:00 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.67@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:33:00.327733 6250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:00.320540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.67:56239)
0929 18:33:00.320985 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:00.321003 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:00.321176 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:00.321705 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:00.321707 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:00.321958 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:33:00.322094 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:00.322102 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:00.323600 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:00.323603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:00.324210 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:00.324217 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:00.324777 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:00.324780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:00.324845 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:00.325473 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:33:00.325485 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:33:00.327593 (+ 2108us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":126,"threads_started":1}
W20260929 18:33:00.328143 6248 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:33:00.320720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:56239 (local address 127.28.207.126:45245)
0929 18:33:00.320898 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:00.320934 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:00.321176 (+ 242us) server_negotiation.cc:407] Connection header received
0929 18:33:00.321315 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:00.321318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:00.321483 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:00.321620 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:00.322259 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:00.323471 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:00.324376 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:00.324643 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:00.324741 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:00.327953 (+ 3212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.67:56239: BlockingRecv error: recv got EOF from 127.28.207.67:56239 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 18:33:00 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.65@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:33:00 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.65@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:33:00.849085 6251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:00.842225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.65:40909)
0929 18:33:00.842520 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:00.842534 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:00.842634 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:00.843173 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:00.843176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:00.843377 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:33:00.843481 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:00.843486 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:00.844493 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:00.844496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:00.845221 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:00.845231 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:00.845845 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:00.845849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:00.845923 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:00.846496 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:33:00.846509 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:33:00.848865 (+ 2356us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":93,"threads_started":1}
W20260929 18:33:00.849248 6252 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:33:00.842337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:40909 (local address 127.28.207.126:45245)
0929 18:33:00.842803 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:00.842807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:00.842823 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:33:00.842916 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:00.842919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:00.842973 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:00.843082 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:00.843734 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:00.844346 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:00.845385 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:00.845699 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:00.845802 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:00.849088 (+ 3286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.65:40909: BlockingRecv error: recv got EOF from 127.28.207.65:40909 (error 108)
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":144,"threads_started":1}
Sep 29 18:33:01 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.66@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 18:33:01 dist-test-slave-qcpv krb5kdc[5194](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.28.207.66@KRBTEST.COM for oryx/127.28.207.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 18:33:01.122344 6253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:01.114328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45245 (local address 127.28.207.66:57219)
0929 18:33:01.114966 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:01.114981 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:01.115185 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:01.115481 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:01.115484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:01.115851 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:33:01.115963 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:01.115977 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:01.116937 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:01.116943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:01.117605 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:01.117614 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:01.118135 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:01.118139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:01.118225 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:01.118974 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:33:01.118987 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:33:01.122117 (+ 3130us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.28.207.126:45245: Server oryx/127.28.207.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":141,"threads_started":1}
W20260929 18:33:01.122524 6252 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:33:01.114566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:57219 (local address 127.28.207.126:45245)
0929 18:33:01.114824 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:01.114831 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:01.115114 (+ 283us) server_negotiation.cc:407] Connection header received
0929 18:33:01.115298 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:01.115302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:01.115359 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:01.115464 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:01.116131 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:01.116800 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:01.117744 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:01.117995 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:01.118119 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:01.122343 (+ 4224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.28.207.66:57219: BlockingRecv error: recv got EOF from 127.28.207.66:57219 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 18:33:01.170535 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 5777
I20260929 18:33:01.176925 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 5929
I20260929 18:33:01.184106 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 6061
I20260929 18:33:01.190662 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 5707
2026-09-29T18:33:01Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18495 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T18:33:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:33:01Z Disabled control of system clock
I20260929 18:33:01.226666 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:40585
--webserver_interface=127.28.207.126
--webserver_port=0
--builtin_ntp_servers=127.28.207.84:43921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:40585
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 18:33:01.392158 6257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:33:01.392485 6257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:33:01.392524 6257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:33:01.398058 6257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:33:01.398196 6257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:33:01.398214 6257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:33:01.398234 6257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:33:01.398252 6257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:33:01.405318 6257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:43921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:40585
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:40585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:33:01.407058 6257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:33:01.408778 6257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:33:01.418527 6263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:01.418540 6265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:01.418922 6262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:01.419126 6257 server_base.cc:1034] running on GCE node
I20260929 18:33:01.419864 6257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:33:01.421123 6257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:33:01.422367 6257 hybrid_clock.cc:648] HybridClock initialized: now 1790706781422328 us; error 67 us; skew 500 ppm
I20260929 18:33:01.424958 6257 webserver.cc:460] Webserver started at http://127.28.207.126:39327/ using document root <none> and password file <none>
I20260929 18:33:01.425621 6257 fs_manager.cc:359] Metadata directory not provided
I20260929 18:33:01.425719 6257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:33:01.425961 6257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:33:01.428473 6257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "0e5ca930ab534ec89d28e50dbd88e9bd"
format_stamp: "Formatted at 2026-09-29 18:33:01 on dist-test-slave-qcpv"
server_key: "a42ca5bb2feb9a62b151646de888df3a"
server_key_iv: "3374f01bf3e9ed3aaf9f63476dcbfed6"
server_key_version: "encryptionkey@0"
I20260929 18:33:01.429087 6257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "0e5ca930ab534ec89d28e50dbd88e9bd"
format_stamp: "Formatted at 2026-09-29 18:33:01 on dist-test-slave-qcpv"
server_key: "a42ca5bb2feb9a62b151646de888df3a"
server_key_iv: "3374f01bf3e9ed3aaf9f63476dcbfed6"
server_key_version: "encryptionkey@0"
I20260929 18:33:01.433143 6257 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:33:01.436342 6271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:01.438198 6257 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260929 18:33:01.438390 6257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "0e5ca930ab534ec89d28e50dbd88e9bd"
format_stamp: "Formatted at 2026-09-29 18:33:01 on dist-test-slave-qcpv"
server_key: "a42ca5bb2feb9a62b151646de888df3a"
server_key_iv: "3374f01bf3e9ed3aaf9f63476dcbfed6"
server_key_version: "encryptionkey@0"
I20260929 18:33:01.438650 6257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:33:01.466106 6257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:33:01.466910 6257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:33:01.467127 6257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:33:01.475508 6257 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:40585
I20260929 18:33:01.475816 6323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:40585 every 8 connection(s)
I20260929 18:33:01.476990 6257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:33:01.477869 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 6257
I20260929 18:33:01.478045 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:33:01.478610 29501 external_mini_cluster.cc:1428] Setting key 8e068f9105c1b0489b7b4e47c2a2f510
I20260929 18:33:01.482378 6324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:33:01.492395 6324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e5ca930ab534ec89d28e50dbd88e9bd: Bootstrap starting.
I20260929 18:33:01.493379 6327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:01.480700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59746 (local address 127.28.207.126:40585)
0929 18:33:01.481543 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:01.481569 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:01.481627 (+ 58us) server_negotiation.cc:407] Connection header received
0929 18:33:01.482685 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:01.482713 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:01.483374 (+ 661us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:01.483932 (+ 558us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:01.484617 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:01.486359 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:01.487629 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:01.488485 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:01.488706 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:01.490004 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:01.490011 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:01.490035 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:33:01.490757 (+ 722us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:01.490884 (+ 127us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:33:01.490924 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:01.491052 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:01.491274 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 18:33:01.491819 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":132,"threads_started":1}
I20260929 18:33:01.495821 6324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e5ca930ab534ec89d28e50dbd88e9bd: Neither blocks nor log segments found. Creating new log.
I20260929 18:33:01.496817 6324 log.cc:824] T 00000000000000000000000000000000 P 0e5ca930ab534ec89d28e50dbd88e9bd: Log is configured to *not* fsync() on all Append() calls
I20260929 18:33:01.499680 6324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e5ca930ab534ec89d28e50dbd88e9bd: No bootstrap required, opened a new log
I20260929 18:33:01.504521 6324 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e5ca930ab534ec89d28e50dbd88e9bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e5ca930ab534ec89d28e50dbd88e9bd" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 40585 } }
I20260929 18:33:01.504923 6324 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e5ca930ab534ec89d28e50dbd88e9bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:33:01.505019 6324 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e5ca930ab534ec89d28e50dbd88e9bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e5ca930ab534ec89d28e50dbd88e9bd, State: Initialized, Role: FOLLOWER
I20260929 18:33:01.505630 6324 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e5ca930ab534ec89d28e50dbd88e9bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e5ca930ab534ec89d28e50dbd88e9bd" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 40585 } }
I20260929 18:33:01.505832 6324 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e5ca930ab534ec89d28e50dbd88e9bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:33:01.505941 6324 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e5ca930ab534ec89d28e50dbd88e9bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:33:01.506098 6324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e5ca930ab534ec89d28e50dbd88e9bd [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:33:01.507501 6324 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e5ca930ab534ec89d28e50dbd88e9bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e5ca930ab534ec89d28e50dbd88e9bd" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 40585 } }
I20260929 18:33:01.508016 6324 leader_election.cc:302] T 00000000000000000000000000000000 P 0e5ca930ab534ec89d28e50dbd88e9bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e5ca930ab534ec89d28e50dbd88e9bd; no voters:
I20260929 18:33:01.508628 6324 leader_election.cc:288] T 00000000000000000000000000000000 P 0e5ca930ab534ec89d28e50dbd88e9bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:33:01.508848 6329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e5ca930ab534ec89d28e50dbd88e9bd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:33:01.509230 6329 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e5ca930ab534ec89d28e50dbd88e9bd [term 1 LEADER]: Becoming Leader. State: Replica: 0e5ca930ab534ec89d28e50dbd88e9bd, State: Running, Role: LEADER
I20260929 18:33:01.509652 6329 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e5ca930ab534ec89d28e50dbd88e9bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e5ca930ab534ec89d28e50dbd88e9bd" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 40585 } }
I20260929 18:33:01.510484 6324 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e5ca930ab534ec89d28e50dbd88e9bd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:33:01.511754 6331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e5ca930ab534ec89d28e50dbd88e9bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e5ca930ab534ec89d28e50dbd88e9bd. Latest consensus state: current_term: 1 leader_uuid: "0e5ca930ab534ec89d28e50dbd88e9bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e5ca930ab534ec89d28e50dbd88e9bd" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 40585 } } }
I20260929 18:33:01.511912 6331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e5ca930ab534ec89d28e50dbd88e9bd [sys.catalog]: This master's current role is: LEADER
I20260929 18:33:01.512276 6330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e5ca930ab534ec89d28e50dbd88e9bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e5ca930ab534ec89d28e50dbd88e9bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e5ca930ab534ec89d28e50dbd88e9bd" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 40585 } } }
I20260929 18:33:01.512370 6330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e5ca930ab534ec89d28e50dbd88e9bd [sys.catalog]: This master's current role is: LEADER
I20260929 18:33:01.512475 6338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:33:01.515589 6338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:33:01.521522 6338 catalog_manager.cc:1347] Generated new cluster ID: a767754e557e41bcaff8f865f8972987
I20260929 18:33:01.521623 6338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:33:01.533677 6338 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:33:01.535643 6338 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:33:01.547953 6338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e5ca930ab534ec89d28e50dbd88e9bd: Generated new TSK 0
I20260929 18:33:01.548816 6338 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:33:01.552327 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.65:0
--local_ip_for_outbound_sockets=127.28.207.65
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:40585
--builtin_ntp_servers=127.28.207.84:43921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 18:33:01.683735 6348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:33:01.684082 6348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:33:01.684152 6348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:33:01.688591 6348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:33:01.688688 6348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:33:01.688778 6348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.65
I20260929 18:33:01.693934 6348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:43921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:40585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.65
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:33:01.695185 6348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:33:01.696568 6348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:33:01.708056 6356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:01.708109 6354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:01.708110 6353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:01.708359 6348 server_base.cc:1034] running on GCE node
I20260929 18:33:01.709149 6348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:33:01.710054 6348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:33:01.711357 6348 hybrid_clock.cc:648] HybridClock initialized: now 1790706781711323 us; error 50 us; skew 500 ppm
I20260929 18:33:01.713802 6348 webserver.cc:460] Webserver started at http://127.28.207.65:42033/ using document root <none> and password file <none>
I20260929 18:33:01.714538 6348 fs_manager.cc:359] Metadata directory not provided
I20260929 18:33:01.714637 6348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:33:01.714974 6348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:33:01.717285 6348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/data/instance:
uuid: "83eea9e2efd24448aebd5258ac59e7ea"
format_stamp: "Formatted at 2026-09-29 18:33:01 on dist-test-slave-qcpv"
server_key: "cc00845c44111e6a08744251455cd648"
server_key_iv: "faba858d04b458608834b97c5308eb44"
server_key_version: "encryptionkey@0"
I20260929 18:33:01.717852 6348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance:
uuid: "83eea9e2efd24448aebd5258ac59e7ea"
format_stamp: "Formatted at 2026-09-29 18:33:01 on dist-test-slave-qcpv"
server_key: "cc00845c44111e6a08744251455cd648"
server_key_iv: "faba858d04b458608834b97c5308eb44"
server_key_version: "encryptionkey@0"
I20260929 18:33:01.722925 6348 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 18:33:01.726591 6362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:01.728439 6348 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:33:01.728636 6348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/wal
uuid: "83eea9e2efd24448aebd5258ac59e7ea"
format_stamp: "Formatted at 2026-09-29 18:33:01 on dist-test-slave-qcpv"
server_key: "cc00845c44111e6a08744251455cd648"
server_key_iv: "faba858d04b458608834b97c5308eb44"
server_key_version: "encryptionkey@0"
I20260929 18:33:01.728780 6348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:33:01.741709 6348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:33:01.742571 6348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:33:01.742818 6348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:33:01.743505 6348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:33:01.744629 6348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:33:01.744704 6348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:01.744768 6348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:33:01.744822 6348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:01.757169 6348 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.65:36419
I20260929 18:33:01.757341 6475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.65:36419 every 8 connection(s)
I20260929 18:33:01.758363 6348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
I20260929 18:33:01.762308 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 6348
I20260929 18:33:01.762439 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance
I20260929 18:33:01.762763 29501 external_mini_cluster.cc:1428] Setting key e62aae766e3b3440225e687b6f76fc62
I20260929 18:33:01.765645 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.66:0
--local_ip_for_outbound_sockets=127.28.207.66
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:40585
--builtin_ntp_servers=127.28.207.84:43921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 18:33:01.772572 6327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:01.761250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:43371 (local address 127.28.207.126:40585)
0929 18:33:01.761424 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:01.761428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:01.762911 (+ 1483us) server_negotiation.cc:407] Connection header received
0929 18:33:01.764753 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:01.764755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:01.764801 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:01.764876 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:01.767044 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:01.767994 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:01.769363 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:01.769779 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:01.769908 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:01.771767 (+ 1859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:01.771773 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:01.771779 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:33:01.771803 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:01.771840 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:33:01.771854 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:01.771965 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:01.772196 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 18:33:01.772308 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 18:33:01.773756 6478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:01.761702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:40585 (local address 127.28.207.65:43371)
0929 18:33:01.762653 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:01.762715 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:01.764372 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:01.765458 (+ 1086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:01.765476 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:01.765656 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:33:01.766809 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:01.766844 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:01.768180 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:01.768184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:01.769204 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:01.769216 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:01.770144 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:01.770150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:01.770406 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:01.771342 (+ 936us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:33:01.771344 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:33:01.771408 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:33:01.771531 (+ 123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:33:01.771988 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:33:01.771996 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:33:01.772002 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:33:01.772136 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 18:33:01.772508 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":145,"threads_started":1}
I20260929 18:33:01.775835 6476 heartbeater.cc:347] Connected to a master server at 127.28.207.126:40585
I20260929 18:33:01.776192 6476 heartbeater.cc:464] Registering TS with master...
I20260929 18:33:01.776973 6476 heartbeater.cc:511] Master 127.28.207.126:40585 requested a full tablet report, sending...
I20260929 18:33:01.779637 6286 ts_manager.cc:194] Registered new tserver with Master: 83eea9e2efd24448aebd5258ac59e7ea (127.28.207.65:36419)
I20260929 18:33:01.781317 6286 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.207.65:43371
W20260929 18:33:01.880481 6479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:33:01.880780 6479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:33:01.880818 6479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:33:01.885313 6479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:33:01.885393 6479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:33:01.885452 6479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.66
I20260929 18:33:01.890389 6479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:43921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:40585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.66
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:33:01.891755 6479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:33:01.893164 6479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:33:01.903991 6484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:01.904037 6485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:01.904161 6487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:01.904654 6479 server_base.cc:1034] running on GCE node
I20260929 18:33:01.905275 6479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:33:01.906076 6479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:33:01.907299 6479 hybrid_clock.cc:648] HybridClock initialized: now 1790706781907275 us; error 56 us; skew 500 ppm
I20260929 18:33:01.909538 6479 webserver.cc:460] Webserver started at http://127.28.207.66:41745/ using document root <none> and password file <none>
I20260929 18:33:01.910163 6479 fs_manager.cc:359] Metadata directory not provided
I20260929 18:33:01.910264 6479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:33:01.910511 6479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:33:01.912863 6479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/data/instance:
uuid: "c516367f2fa14f25aae77f63565dbb40"
format_stamp: "Formatted at 2026-09-29 18:33:01 on dist-test-slave-qcpv"
server_key: "d26ab20072639fd4f8c5800ae27fd322"
server_key_iv: "8ca92463fba00fcddc64ee84c181635d"
server_key_version: "encryptionkey@0"
I20260929 18:33:01.913461 6479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance:
uuid: "c516367f2fa14f25aae77f63565dbb40"
format_stamp: "Formatted at 2026-09-29 18:33:01 on dist-test-slave-qcpv"
server_key: "d26ab20072639fd4f8c5800ae27fd322"
server_key_iv: "8ca92463fba00fcddc64ee84c181635d"
server_key_version: "encryptionkey@0"
I20260929 18:33:01.917675 6479 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:33:01.920837 6493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:01.922525 6479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 18:33:01.922724 6479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/wal
uuid: "c516367f2fa14f25aae77f63565dbb40"
format_stamp: "Formatted at 2026-09-29 18:33:01 on dist-test-slave-qcpv"
server_key: "d26ab20072639fd4f8c5800ae27fd322"
server_key_iv: "8ca92463fba00fcddc64ee84c181635d"
server_key_version: "encryptionkey@0"
I20260929 18:33:01.922894 6479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:33:01.967701 6479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:33:01.968468 6479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:33:01.968683 6479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:33:01.969300 6479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:33:01.970331 6479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:33:01.970512 6479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:01.970633 6479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:33:01.970688 6479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:01.981462 6479 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.66:35021
I20260929 18:33:01.981482 6606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.66:35021 every 8 connection(s)
I20260929 18:33:01.982448 6479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
I20260929 18:33:01.985404 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 6479
I20260929 18:33:01.985494 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance
I20260929 18:33:01.985818 29501 external_mini_cluster.cc:1428] Setting key f840982a5849b5fed2efaa20c855f908
I20260929 18:33:01.988145 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.67:0
--local_ip_for_outbound_sockets=127.28.207.67
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:40585
--builtin_ntp_servers=127.28.207.84:43921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 18:33:01.996506 6327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:01.984775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:41275 (local address 127.28.207.126:40585)
0929 18:33:01.984921 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:01.984925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:01.986835 (+ 1910us) server_negotiation.cc:407] Connection header received
0929 18:33:01.988720 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:01.988728 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:01.988800 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:01.988922 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:01.990792 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:01.991791 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:01.993131 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:01.993565 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:01.993704 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:01.995750 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:01.995755 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:01.995760 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:33:01.995783 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:01.995817 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:33:01.995833 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:01.995943 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:01.996194 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 18:33:01.996282 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 18:33:01.997749 6609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:01.985428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:40585 (local address 127.28.207.66:41275)
0929 18:33:01.986583 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:01.986650 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:01.988405 (+ 1755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:01.989211 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:01.989221 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:01.989542 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:33:01.990517 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:01.990551 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:01.991980 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:01.991983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:01.992992 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:01.993004 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:01.993876 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:01.993896 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:01.994184 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:01.995239 (+ 1055us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:33:01.995241 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:33:01.995310 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:33:01.995413 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:33:01.995980 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:33:01.995989 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:33:01.995995 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:33:01.996176 (+ 181us) client_negotiation.cc:240] Negotiation successful
0929 18:33:01.996533 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":216,"threads_started":1}
I20260929 18:33:01.999691 6607 heartbeater.cc:347] Connected to a master server at 127.28.207.126:40585
I20260929 18:33:02.000032 6607 heartbeater.cc:464] Registering TS with master...
I20260929 18:33:02.000681 6607 heartbeater.cc:511] Master 127.28.207.126:40585 requested a full tablet report, sending...
I20260929 18:33:02.002095 6286 ts_manager.cc:194] Registered new tserver with Master: c516367f2fa14f25aae77f63565dbb40 (127.28.207.66:35021)
I20260929 18:33:02.003208 6286 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.207.66:41275
W20260929 18:33:02.118275 6610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:33:02.118579 6610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:33:02.118618 6610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:33:02.122797 6610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:33:02.122896 6610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:33:02.123005 6610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.67
I20260929 18:33:02.128051 6610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:43921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:40585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.67
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:33:02.129189 6610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:33:02.130529 6610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:33:02.144328 6618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:02.144354 6615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:02.144508 6610 server_base.cc:1034] running on GCE node
W20260929 18:33:02.144461 6616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:02.145212 6610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:33:02.145999 6610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:33:02.147199 6610 hybrid_clock.cc:648] HybridClock initialized: now 1790706782147182 us; error 53 us; skew 500 ppm
I20260929 18:33:02.149170 6610 webserver.cc:460] Webserver started at http://127.28.207.67:45191/ using document root <none> and password file <none>
I20260929 18:33:02.149762 6610 fs_manager.cc:359] Metadata directory not provided
I20260929 18:33:02.149856 6610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:33:02.150089 6610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:33:02.152281 6610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/data/instance:
uuid: "1bf4567b2e724577b2f1f591fb3509b0"
format_stamp: "Formatted at 2026-09-29 18:33:02 on dist-test-slave-qcpv"
server_key: "2c009b275c7e7a0de574089c0c0ab4c5"
server_key_iv: "8eb39f4d63b8cbbf98b7e0f61d00bd67"
server_key_version: "encryptionkey@0"
I20260929 18:33:02.152848 6610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance:
uuid: "1bf4567b2e724577b2f1f591fb3509b0"
format_stamp: "Formatted at 2026-09-29 18:33:02 on dist-test-slave-qcpv"
server_key: "2c009b275c7e7a0de574089c0c0ab4c5"
server_key_iv: "8eb39f4d63b8cbbf98b7e0f61d00bd67"
server_key_version: "encryptionkey@0"
I20260929 18:33:02.156687 6610 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:33:02.159869 6624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:02.161403 6610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:33:02.161568 6610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/wal
uuid: "1bf4567b2e724577b2f1f591fb3509b0"
format_stamp: "Formatted at 2026-09-29 18:33:02 on dist-test-slave-qcpv"
server_key: "2c009b275c7e7a0de574089c0c0ab4c5"
server_key_iv: "8eb39f4d63b8cbbf98b7e0f61d00bd67"
server_key_version: "encryptionkey@0"
I20260929 18:33:02.161701 6610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:33:02.184648 6610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:33:02.185405 6610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:33:02.185611 6610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:33:02.186244 6610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:33:02.187201 6610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:33:02.187279 6610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:02.187358 6610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:33:02.187402 6610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:02.198271 6610 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.67:36715
I20260929 18:33:02.198334 6737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.67:36715 every 8 connection(s)
I20260929 18:33:02.199231 6610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
I20260929 18:33:02.207993 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 6610
I20260929 18:33:02.208153 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance
I20260929 18:33:02.208518 29501 external_mini_cluster.cc:1428] Setting key 062ab10d76545027cf5e22b626209eef
I20260929 18:33:02.212472 6327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:02.201863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:47325 (local address 127.28.207.126:40585)
0929 18:33:02.202047 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:02.202052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:02.203572 (+ 1520us) server_negotiation.cc:407] Connection header received
0929 18:33:02.205235 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:02.205239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:02.205298 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:02.205405 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:02.206706 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:02.207369 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:02.209378 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:02.209638 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:02.209778 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:02.211715 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:02.211719 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:02.211723 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:33:02.211750 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:02.211779 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:33:02.211790 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:02.211916 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:02.212274 (+ 358us) server_negotiation.cc:299] Negotiation successful
0929 18:33:02.212348 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 18:33:02.213481 6740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:02.202302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:40585 (local address 127.28.207.67:47325)
0929 18:33:02.203280 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:02.203342 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:02.204915 (+ 1573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:02.205695 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:02.205704 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:02.205812 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:33:02.206484 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:02.206498 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:02.207643 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:02.207649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:02.209194 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:02.209206 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:02.209880 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:02.209885 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:02.210205 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:02.211272 (+ 1067us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:33:02.211274 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:33:02.211319 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:33:02.211436 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:33:02.211955 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:33:02.211964 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:33:02.211973 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 18:33:02.212218 (+ 245us) client_negotiation.cc:240] Negotiation successful
0929 18:33:02.212541 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":129,"threads_started":1}
I20260929 18:33:02.214956 6738 heartbeater.cc:347] Connected to a master server at 127.28.207.126:40585
I20260929 18:33:02.215281 6738 heartbeater.cc:464] Registering TS with master...
I20260929 18:33:02.215937 6738 heartbeater.cc:511] Master 127.28.207.126:40585 requested a full tablet report, sending...
I20260929 18:33:02.217140 6286 ts_manager.cc:194] Registered new tserver with Master: 1bf4567b2e724577b2f1f591fb3509b0 (127.28.207.67:36715)
I20260929 18:33:02.217710 6286 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.207.67:47325
I20260929 18:33:02.223536 29501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 18:33:02.227609 6327 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:33:02.226142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59750 (local address 127.28.207.126:40585)
0929 18:33:02.226394 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:02.226405 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:02.226613 (+ 208us) server_negotiation.cc:407] Connection header received
0929 18:33:02.226752 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:02.226754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:02.226843 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:02.226992 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:02.227458 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59750: BlockingRecv error: recv got EOF from 127.0.0.1:59750 (error 108)
Metrics: {"server-negotiator.queue_time_us":141}
I20260929 18:33:02.227713 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 6348
I20260929 18:33:02.235324 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 6479
I20260929 18:33:02.242269 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 6610
I20260929 18:33:02.247943 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 6257
2026-09-29T18:33:02Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (1054 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T18:33:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:33:02Z Disabled control of system clock
I20260929 18:33:02.280305 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:36721
--webserver_interface=127.28.207.126
--webserver_port=0
--builtin_ntp_servers=127.28.207.84:42105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:36721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 18:33:02.394476 6749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:33:02.394811 6749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:33:02.394877 6749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:33:02.399236 6749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:33:02.399314 6749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:33:02.399330 6749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:33:02.399348 6749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:33:02.399363 6749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:33:02.404570 6749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:42105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:36721
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:36721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:33:02.405794 6749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:33:02.407331 6749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:33:02.417286 6757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:02.417263 6754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:02.417263 6755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:02.417681 6749 server_base.cc:1034] running on GCE node
I20260929 18:33:02.418429 6749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:33:02.419721 6749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:33:02.423013 6749 hybrid_clock.cc:648] HybridClock initialized: now 1790706782422867 us; error 162 us; skew 500 ppm
I20260929 18:33:02.425487 6749 webserver.cc:460] Webserver started at http://127.28.207.126:41335/ using document root <none> and password file <none>
I20260929 18:33:02.426139 6749 fs_manager.cc:359] Metadata directory not provided
I20260929 18:33:02.426215 6749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:33:02.426440 6749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:33:02.428722 6749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "904c0989349d46f0b18ff9a0704ed767"
format_stamp: "Formatted at 2026-09-29 18:33:02 on dist-test-slave-qcpv"
server_key: "9d6a8a0fcec3a364d5cc8fa098e3f585"
server_key_iv: "1f3b6e617ce1454bc272bcdbf36676fe"
server_key_version: "encryptionkey@0"
I20260929 18:33:02.429325 6749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "904c0989349d46f0b18ff9a0704ed767"
format_stamp: "Formatted at 2026-09-29 18:33:02 on dist-test-slave-qcpv"
server_key: "9d6a8a0fcec3a364d5cc8fa098e3f585"
server_key_iv: "1f3b6e617ce1454bc272bcdbf36676fe"
server_key_version: "encryptionkey@0"
I20260929 18:33:02.433442 6749 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:33:02.436339 6763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:02.437980 6749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 18:33:02.438158 6749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "904c0989349d46f0b18ff9a0704ed767"
format_stamp: "Formatted at 2026-09-29 18:33:02 on dist-test-slave-qcpv"
server_key: "9d6a8a0fcec3a364d5cc8fa098e3f585"
server_key_iv: "1f3b6e617ce1454bc272bcdbf36676fe"
server_key_version: "encryptionkey@0"
I20260929 18:33:02.438292 6749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:33:02.443161 6749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:33:02.443784 6749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:33:02.443980 6749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:33:02.451679 6749 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:36721
I20260929 18:33:02.451828 6815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:36721 every 8 connection(s)
I20260929 18:33:02.452713 6749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:33:02.457266 6816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:33:02.457752 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 6749
I20260929 18:33:02.457914 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:33:02.458238 29501 external_mini_cluster.cc:1428] Setting key b740a025e4e9894effe6a58ab2c9dfaf
I20260929 18:33:02.465322 6819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:02.460177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35834 (local address 127.28.207.126:36721)
0929 18:33:02.461154 (+ 977us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:02.461171 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:02.461210 (+ 39us) server_negotiation.cc:407] Connection header received
0929 18:33:02.462027 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:02.462045 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:02.462439 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:02.462777 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:02.463925 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:02.463930 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:02.463942 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:33:02.463952 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:02.464013 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:33:02.464030 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:02.464153 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:02.464311 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 18:33:02.464599 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":188,"threads_started":1}
I20260929 18:33:02.465700 6816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 904c0989349d46f0b18ff9a0704ed767: Bootstrap starting.
I20260929 18:33:02.469144 6816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 904c0989349d46f0b18ff9a0704ed767: Neither blocks nor log segments found. Creating new log.
I20260929 18:33:02.470356 6816 log.cc:824] T 00000000000000000000000000000000 P 904c0989349d46f0b18ff9a0704ed767: Log is configured to *not* fsync() on all Append() calls
I20260929 18:33:02.473290 6816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 904c0989349d46f0b18ff9a0704ed767: No bootstrap required, opened a new log
I20260929 18:33:02.478152 6816 raft_consensus.cc:348] T 00000000000000000000000000000000 P 904c0989349d46f0b18ff9a0704ed767 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "904c0989349d46f0b18ff9a0704ed767" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 36721 } }
I20260929 18:33:02.478507 6816 raft_consensus.cc:374] T 00000000000000000000000000000000 P 904c0989349d46f0b18ff9a0704ed767 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:33:02.478587 6816 raft_consensus.cc:729] T 00000000000000000000000000000000 P 904c0989349d46f0b18ff9a0704ed767 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 904c0989349d46f0b18ff9a0704ed767, State: Initialized, Role: FOLLOWER
I20260929 18:33:02.479212 6816 consensus_queue.cc:260] T 00000000000000000000000000000000 P 904c0989349d46f0b18ff9a0704ed767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "904c0989349d46f0b18ff9a0704ed767" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 36721 } }
I20260929 18:33:02.479372 6816 raft_consensus.cc:388] T 00000000000000000000000000000000 P 904c0989349d46f0b18ff9a0704ed767 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:33:02.479472 6816 raft_consensus.cc:482] T 00000000000000000000000000000000 P 904c0989349d46f0b18ff9a0704ed767 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:33:02.479591 6816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 904c0989349d46f0b18ff9a0704ed767 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:33:02.480798 6816 raft_consensus.cc:504] T 00000000000000000000000000000000 P 904c0989349d46f0b18ff9a0704ed767 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "904c0989349d46f0b18ff9a0704ed767" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 36721 } }
I20260929 18:33:02.481248 6816 leader_election.cc:302] T 00000000000000000000000000000000 P 904c0989349d46f0b18ff9a0704ed767 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 904c0989349d46f0b18ff9a0704ed767; no voters:
I20260929 18:33:02.481711 6816 leader_election.cc:288] T 00000000000000000000000000000000 P 904c0989349d46f0b18ff9a0704ed767 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:33:02.481954 6821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 904c0989349d46f0b18ff9a0704ed767 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:33:02.482353 6821 raft_consensus.cc:686] T 00000000000000000000000000000000 P 904c0989349d46f0b18ff9a0704ed767 [term 1 LEADER]: Becoming Leader. State: Replica: 904c0989349d46f0b18ff9a0704ed767, State: Running, Role: LEADER
I20260929 18:33:02.482792 6821 consensus_queue.cc:237] T 00000000000000000000000000000000 P 904c0989349d46f0b18ff9a0704ed767 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "904c0989349d46f0b18ff9a0704ed767" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 36721 } }
I20260929 18:33:02.483659 6816 sys_catalog.cc:564] T 00000000000000000000000000000000 P 904c0989349d46f0b18ff9a0704ed767 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:33:02.484990 6822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 904c0989349d46f0b18ff9a0704ed767 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "904c0989349d46f0b18ff9a0704ed767" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "904c0989349d46f0b18ff9a0704ed767" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 36721 } } }
I20260929 18:33:02.485149 6822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 904c0989349d46f0b18ff9a0704ed767 [sys.catalog]: This master's current role is: LEADER
I20260929 18:33:02.485409 6823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 904c0989349d46f0b18ff9a0704ed767 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 904c0989349d46f0b18ff9a0704ed767. Latest consensus state: current_term: 1 leader_uuid: "904c0989349d46f0b18ff9a0704ed767" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "904c0989349d46f0b18ff9a0704ed767" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 36721 } } }
I20260929 18:33:02.485518 6823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 904c0989349d46f0b18ff9a0704ed767 [sys.catalog]: This master's current role is: LEADER
I20260929 18:33:02.485807 6830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:33:02.489634 6830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:33:02.496825 6830 catalog_manager.cc:1347] Generated new cluster ID: 09eae60078e7477abecc0fc6a2305152
I20260929 18:33:02.497071 6830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:33:02.523900 6830 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:33:02.524415 6830 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:33:02.533281 6830 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 904c0989349d46f0b18ff9a0704ed767: Generated new TSK 0
I20260929 18:33:02.534260 6830 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:33:02.549477 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.65:0
--local_ip_for_outbound_sockets=127.28.207.65
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:36721
--builtin_ntp_servers=127.28.207.84:42105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 18:33:02.699467 6840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:33:02.699940 6840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:33:02.700037 6840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:33:02.706964 6840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:33:02.707298 6840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:33:02.707451 6840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.65
I20260929 18:33:02.715345 6840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:42105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:36721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.65
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:33:02.717031 6840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:33:02.719049 6840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:33:02.731729 6845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:02.731719 6848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:02.731719 6846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:02.732211 6840 server_base.cc:1034] running on GCE node
I20260929 18:33:02.732810 6840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:33:02.733677 6840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:33:02.734952 6840 hybrid_clock.cc:648] HybridClock initialized: now 1790706782734912 us; error 52 us; skew 500 ppm
I20260929 18:33:02.737511 6840 webserver.cc:460] Webserver started at http://127.28.207.65:43867/ using document root <none> and password file <none>
I20260929 18:33:02.738226 6840 fs_manager.cc:359] Metadata directory not provided
I20260929 18:33:02.738309 6840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:33:02.738523 6840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:33:02.741530 6840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/data/instance:
uuid: "3115da278a3d4923a4e23731507d5a48"
format_stamp: "Formatted at 2026-09-29 18:33:02 on dist-test-slave-qcpv"
server_key: "3261e1afb87e296b0dd88f6e0cbeecfe"
server_key_iv: "dbcabfd6e24d2e2cf7d302330ec990fa"
server_key_version: "encryptionkey@0"
I20260929 18:33:02.742316 6840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance:
uuid: "3115da278a3d4923a4e23731507d5a48"
format_stamp: "Formatted at 2026-09-29 18:33:02 on dist-test-slave-qcpv"
server_key: "3261e1afb87e296b0dd88f6e0cbeecfe"
server_key_iv: "dbcabfd6e24d2e2cf7d302330ec990fa"
server_key_version: "encryptionkey@0"
I20260929 18:33:02.747718 6840 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 18:33:02.751950 6854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:02.753981 6840 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:33:02.754273 6840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/wal
uuid: "3115da278a3d4923a4e23731507d5a48"
format_stamp: "Formatted at 2026-09-29 18:33:02 on dist-test-slave-qcpv"
server_key: "3261e1afb87e296b0dd88f6e0cbeecfe"
server_key_iv: "dbcabfd6e24d2e2cf7d302330ec990fa"
server_key_version: "encryptionkey@0"
I20260929 18:33:02.754449 6840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:33:02.761163 6840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:33:02.762006 6840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:33:02.762247 6840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:33:02.763046 6840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:33:02.764207 6840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:33:02.764262 6840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:02.764335 6840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:33:02.764370 6840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:02.777889 6840 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.65:36247
I20260929 18:33:02.777923 6967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.65:36247 every 8 connection(s)
I20260929 18:33:02.779296 6840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
I20260929 18:33:02.787230 6819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:02.781613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:50807 (local address 127.28.207.126:36721)
0929 18:33:02.781815 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:02.781821 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:02.783112 (+ 1291us) server_negotiation.cc:407] Connection header received
0929 18:33:02.784643 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:02.784647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:02.784760 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:02.784889 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:02.786536 (+ 1647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:02.786543 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:02.786546 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:33:02.786557 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:02.786576 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:33:02.786584 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:02.786673 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:02.786994 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 18:33:02.787060 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 18:33:02.788003 6970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:02.782065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36721 (local address 127.28.207.65:50807)
0929 18:33:02.782917 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:02.782962 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:02.784412 (+ 1450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:02.785180 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:02.785191 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:02.785365 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:33:02.786222 (+ 857us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:33:02.786224 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:33:02.786276 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:33:02.786355 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:33:02.786681 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:33:02.786688 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:33:02.786693 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:33:02.786842 (+ 149us) client_negotiation.cc:240] Negotiation successful
0929 18:33:02.787076 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":148,"threads_started":1}
I20260929 18:33:02.788712 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 6840
I20260929 18:33:02.788846 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance
I20260929 18:33:02.789283 29501 external_mini_cluster.cc:1428] Setting key 184bcb859254034127f2a5442694c6d4
I20260929 18:33:02.789672 6968 heartbeater.cc:347] Connected to a master server at 127.28.207.126:36721
I20260929 18:33:02.789960 6968 heartbeater.cc:464] Registering TS with master...
I20260929 18:33:02.790570 6968 heartbeater.cc:511] Master 127.28.207.126:36721 requested a full tablet report, sending...
I20260929 18:33:02.791965 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.66:0
--local_ip_for_outbound_sockets=127.28.207.66
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:36721
--builtin_ntp_servers=127.28.207.84:42105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 18:33:02.792459 6774 ts_manager.cc:194] Registered new tserver with Master: 3115da278a3d4923a4e23731507d5a48 (127.28.207.65:36247)
W20260929 18:33:02.916841 6971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:33:02.917138 6971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:33:02.917179 6971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:33:02.921259 6971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:33:02.921330 6971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:33:02.921389 6971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.66
I20260929 18:33:02.926252 6971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:42105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:36721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.66
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:33:02.927532 6971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:33:02.928972 6971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:33:02.939431 6977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:02.939431 6976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:02.939734 6971 server_base.cc:1034] running on GCE node
W20260929 18:33:02.939433 6979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:02.940644 6971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:33:02.941359 6971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:33:02.944661 6971 hybrid_clock.cc:648] HybridClock initialized: now 1790706782944601 us; error 80 us; skew 500 ppm
I20260929 18:33:02.947283 6971 webserver.cc:460] Webserver started at http://127.28.207.66:39211/ using document root <none> and password file <none>
I20260929 18:33:02.947994 6971 fs_manager.cc:359] Metadata directory not provided
I20260929 18:33:02.948074 6971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:33:02.948326 6971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:33:02.950717 6971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/data/instance:
uuid: "1cb14fc132754669a17164a8475ead39"
format_stamp: "Formatted at 2026-09-29 18:33:02 on dist-test-slave-qcpv"
server_key: "b17701556559506586816e2102113af6"
server_key_iv: "7683d8c1abb84591a583b0c49b79d68d"
server_key_version: "encryptionkey@0"
I20260929 18:33:02.951460 6971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance:
uuid: "1cb14fc132754669a17164a8475ead39"
format_stamp: "Formatted at 2026-09-29 18:33:02 on dist-test-slave-qcpv"
server_key: "b17701556559506586816e2102113af6"
server_key_iv: "7683d8c1abb84591a583b0c49b79d68d"
server_key_version: "encryptionkey@0"
I20260929 18:33:02.956079 6971 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:33:02.959908 6985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:02.962493 6971 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260929 18:33:02.962703 6971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/wal
uuid: "1cb14fc132754669a17164a8475ead39"
format_stamp: "Formatted at 2026-09-29 18:33:02 on dist-test-slave-qcpv"
server_key: "b17701556559506586816e2102113af6"
server_key_iv: "7683d8c1abb84591a583b0c49b79d68d"
server_key_version: "encryptionkey@0"
I20260929 18:33:02.962960 6971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:33:02.969333 6971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:33:02.970472 6971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:33:02.970779 6971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:33:02.971645 6971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:33:02.972923 6971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:33:02.973001 6971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:02.973052 6971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:33:02.973099 6971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:02.985658 6971 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.66:34737
I20260929 18:33:02.985704 7098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.66:34737 every 8 connection(s)
I20260929 18:33:02.987118 6971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
I20260929 18:33:02.991428 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 6971
I20260929 18:33:02.991606 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance
I20260929 18:33:02.991916 29501 external_mini_cluster.cc:1428] Setting key 9b5d2b7f4f737a4facab440b283b10dc
I20260929 18:33:02.994156 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.67:0
--local_ip_for_outbound_sockets=127.28.207.67
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:36721
--builtin_ntp_servers=127.28.207.84:42105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 18:33:02.995720 6819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:02.989372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:32893 (local address 127.28.207.126:36721)
0929 18:33:02.989567 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:02.989571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:02.991546 (+ 1975us) server_negotiation.cc:407] Connection header received
0929 18:33:02.993403 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:02.993407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:02.993465 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:02.993587 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:02.995147 (+ 1560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:02.995151 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:02.995154 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:33:02.995172 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:02.995200 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:33:02.995209 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:02.995322 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:02.995477 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 18:33:02.995537 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 18:33:02.996872 7101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:02.989726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36721 (local address 127.28.207.66:32893)
0929 18:33:02.991325 (+ 1599us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:02.991386 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:02.993115 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:02.993811 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:02.993820 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:02.993941 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:33:02.994782 (+ 841us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:33:02.994783 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:33:02.994836 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:33:02.994987 (+ 151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:33:02.995298 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:33:02.995307 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:33:02.995314 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 18:33:02.995547 (+ 233us) client_negotiation.cc:240] Negotiation successful
0929 18:33:02.995874 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":172,"threads_started":1}
I20260929 18:33:02.998417 7099 heartbeater.cc:347] Connected to a master server at 127.28.207.126:36721
I20260929 18:33:02.998785 7099 heartbeater.cc:464] Registering TS with master...
I20260929 18:33:02.999449 7099 heartbeater.cc:511] Master 127.28.207.126:36721 requested a full tablet report, sending...
I20260929 18:33:03.000993 6774 ts_manager.cc:194] Registered new tserver with Master: 1cb14fc132754669a17164a8475ead39 (127.28.207.66:34737)
W20260929 18:33:03.115021 7102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:33:03.115332 7102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:33:03.115374 7102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:33:03.119535 7102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:33:03.119602 7102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:33:03.119660 7102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.67
I20260929 18:33:03.124857 7102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:42105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:36721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.67
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:33:03.126089 7102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:33:03.127629 7102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:33:03.139119 7110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:03.139304 7108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:03.139698 7107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:03.139770 7102 server_base.cc:1034] running on GCE node
I20260929 18:33:03.140343 7102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:33:03.141253 7102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:33:03.142589 7102 hybrid_clock.cc:648] HybridClock initialized: now 1790706783142545 us; error 59 us; skew 500 ppm
I20260929 18:33:03.145309 7102 webserver.cc:460] Webserver started at http://127.28.207.67:41183/ using document root <none> and password file <none>
I20260929 18:33:03.146138 7102 fs_manager.cc:359] Metadata directory not provided
I20260929 18:33:03.146255 7102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:33:03.146536 7102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:33:03.149353 7102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/data/instance:
uuid: "3a7b99a2ee524d38b6652846e73fceda"
format_stamp: "Formatted at 2026-09-29 18:33:03 on dist-test-slave-qcpv"
server_key: "4617b3b9ab17524222fb5d295790374a"
server_key_iv: "2cdc27a3f3472592fa625676551ecb76"
server_key_version: "encryptionkey@0"
I20260929 18:33:03.150023 7102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance:
uuid: "3a7b99a2ee524d38b6652846e73fceda"
format_stamp: "Formatted at 2026-09-29 18:33:03 on dist-test-slave-qcpv"
server_key: "4617b3b9ab17524222fb5d295790374a"
server_key_iv: "2cdc27a3f3472592fa625676551ecb76"
server_key_version: "encryptionkey@0"
I20260929 18:33:03.155169 7102 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 18:33:03.159003 7116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:03.161038 7102 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:33:03.161258 7102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/wal
uuid: "3a7b99a2ee524d38b6652846e73fceda"
format_stamp: "Formatted at 2026-09-29 18:33:03 on dist-test-slave-qcpv"
server_key: "4617b3b9ab17524222fb5d295790374a"
server_key_iv: "2cdc27a3f3472592fa625676551ecb76"
server_key_version: "encryptionkey@0"
I20260929 18:33:03.161437 7102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:33:03.167213 7102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:33:03.168232 7102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:33:03.168526 7102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:33:03.169296 7102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:33:03.170457 7102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:33:03.170538 7102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:03.170624 7102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:33:03.170676 7102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:03.183055 7102 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.67:41329
I20260929 18:33:03.183126 7229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.67:41329 every 8 connection(s)
I20260929 18:33:03.184412 7102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
I20260929 18:33:03.191926 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 7102
I20260929 18:33:03.191984 6819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:03.186719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:57807 (local address 127.28.207.126:36721)
0929 18:33:03.186944 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:03.186949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:03.188318 (+ 1369us) server_negotiation.cc:407] Connection header received
0929 18:33:03.189661 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:03.189670 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:03.189723 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:03.189824 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:03.191316 (+ 1492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:03.191320 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:03.191323 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:33:03.191341 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:03.191373 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:33:03.191384 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:03.191563 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:03.191724 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 18:33:03.191791 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 18:33:03.192071 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance
I20260929 18:33:03.192417 29501 external_mini_cluster.cc:1428] Setting key 6c3d9993813d786808d177037dba1d60
I20260929 18:33:03.192886 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:03.187197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36721 (local address 127.28.207.67:57807)
0929 18:33:03.188117 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:03.188162 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:03.189384 (+ 1222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:03.190122 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:03.190130 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:03.190231 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:33:03.190987 (+ 756us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:33:03.190989 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:33:03.191031 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:33:03.191110 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:33:03.191539 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:33:03.191546 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:33:03.191551 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:33:03.191715 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 18:33:03.191915 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":145,"threads_started":1}
I20260929 18:33:03.194453 7230 heartbeater.cc:347] Connected to a master server at 127.28.207.126:36721
I20260929 18:33:03.194721 7230 heartbeater.cc:464] Registering TS with master...
I20260929 18:33:03.195286 7230 heartbeater.cc:511] Master 127.28.207.126:36721 requested a full tablet report, sending...
I20260929 18:33:03.196626 6774 ts_manager.cc:194] Registered new tserver with Master: 3a7b99a2ee524d38b6652846e73fceda (127.28.207.67:41329)
I20260929 18:33:03.208820 29501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 18:33:03.212659 6819 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:33:03.211401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35844 (local address 127.28.207.126:36721)
0929 18:33:03.211548 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:03.211552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:03.211782 (+ 230us) server_negotiation.cc:407] Connection header received
0929 18:33:03.211894 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:03.211897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:03.211937 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:03.212041 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:03.212451 (+ 410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35844: BlockingRecv error: recv got EOF from 127.0.0.1:35844 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:33:03.213092 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 6840
I20260929 18:33:03.219400 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 6971
I20260929 18:33:03.226179 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 7102
I20260929 18:33:03.233129 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 6749
2026-09-29T18:33:03Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (985 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:33:03 dist-test-slave-qcpv krb5kdc[7240](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:33:03 dist-test-slave-qcpv krb5kdc[7240](info): set up 2 sockets
Sep 29 18:33:03 dist-test-slave-qcpv krb5kdc[7240](info): commencing operation
krb5kdc: starting...
W20260929 18:33:04.324431 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:33:04 dist-test-slave-qcpv krb5kdc[7240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:33:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:33:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.207.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
I20260929 18:33:04.476450 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:43633
--webserver_interface=127.28.207.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:33787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:43633
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:33:04.625576 7253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:33:04.625964 7253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:33:04.626011 7253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:33:04.630498 7253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:33:04.630601 7253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:33:04.630622 7253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:33:04.630641 7253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:33:04.630658 7253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:33:04.637207 7253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:33787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:43633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:43633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:33:04.638731 7253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:33:04.640161 7253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:33:04.651392 7258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:04.651461 7261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:04.651454 7259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:04.651537 7253 server_base.cc:1034] running on GCE node
I20260929 18:33:04.652619 7253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:33:04.654379 7253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:33:04.655884 7253 hybrid_clock.cc:648] HybridClock initialized: now 1790706784655794 us; error 102 us; skew 500 ppm
Sep 29 18:33:04 dist-test-slave-qcpv krb5kdc[7240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706784, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:33:04.667683 7253 init.cc:377] Logged in from keytab as kudu/127.28.207.126@KRBTEST.COM (short username kudu)
I20260929 18:33:04.669046 7253 webserver.cc:460] Webserver started at http://127.28.207.126:39653/ using document root <none> and password file <none>
I20260929 18:33:04.669750 7253 fs_manager.cc:359] Metadata directory not provided
I20260929 18:33:04.669826 7253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:33:04.670042 7253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:33:04.672834 7253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "4c1d2289be0a4c12ac600428f803bff2"
format_stamp: "Formatted at 2026-09-29 18:33:04 on dist-test-slave-qcpv"
server_key: "a0b279d7fb93642fdc3019fb6c1f75ce"
server_key_iv: "76c138af651cf33700da81891e1e827d"
server_key_version: "encryptionkey@0"
I20260929 18:33:04.673511 7253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "4c1d2289be0a4c12ac600428f803bff2"
format_stamp: "Formatted at 2026-09-29 18:33:04 on dist-test-slave-qcpv"
server_key: "a0b279d7fb93642fdc3019fb6c1f75ce"
server_key_iv: "76c138af651cf33700da81891e1e827d"
server_key_version: "encryptionkey@0"
I20260929 18:33:04.678571 7253 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 18:33:04.682343 7268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:04.684443 7253 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:33:04.684640 7253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "4c1d2289be0a4c12ac600428f803bff2"
format_stamp: "Formatted at 2026-09-29 18:33:04 on dist-test-slave-qcpv"
server_key: "a0b279d7fb93642fdc3019fb6c1f75ce"
server_key_iv: "76c138af651cf33700da81891e1e827d"
server_key_version: "encryptionkey@0"
I20260929 18:33:04.684793 7253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:33:04.723246 7253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:33:04.728645 7253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:33:04.728847 7253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:33:04.740381 7253 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:43633
I20260929 18:33:04.740501 7320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:43633 every 8 connection(s)
I20260929 18:33:04.741779 7253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:33:04.746078 7321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:33:04.746768 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 7253
I20260929 18:33:04.746938 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:33:04.747268 29501 external_mini_cluster.cc:1428] Setting key 8a9853fdd1b94e05f61a33d146355fe4
I20260929 18:33:04.753612 7321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c1d2289be0a4c12ac600428f803bff2: Bootstrap starting.
I20260929 18:33:04.756927 7321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c1d2289be0a4c12ac600428f803bff2: Neither blocks nor log segments found. Creating new log.
I20260929 18:33:04.757903 7321 log.cc:824] T 00000000000000000000000000000000 P 4c1d2289be0a4c12ac600428f803bff2: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:33:04 dist-test-slave-qcpv krb5kdc[7240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:33:04.761371 7321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c1d2289be0a4c12ac600428f803bff2: No bootstrap required, opened a new log
I20260929 18:33:04.765759 7321 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c1d2289be0a4c12ac600428f803bff2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1d2289be0a4c12ac600428f803bff2" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43633 } }
I20260929 18:33:04.766098 7321 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c1d2289be0a4c12ac600428f803bff2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:33:04.766196 7321 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c1d2289be0a4c12ac600428f803bff2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1d2289be0a4c12ac600428f803bff2, State: Initialized, Role: FOLLOWER
I20260929 18:33:04.766680 7321 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c1d2289be0a4c12ac600428f803bff2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1d2289be0a4c12ac600428f803bff2" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43633 } }
I20260929 18:33:04.766844 7321 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c1d2289be0a4c12ac600428f803bff2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:33:04.766937 7321 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c1d2289be0a4c12ac600428f803bff2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:33:04.767033 7321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c1d2289be0a4c12ac600428f803bff2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:33:04.768254 7321 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c1d2289be0a4c12ac600428f803bff2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1d2289be0a4c12ac600428f803bff2" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43633 } }
I20260929 18:33:04.768740 7321 leader_election.cc:302] T 00000000000000000000000000000000 P 4c1d2289be0a4c12ac600428f803bff2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c1d2289be0a4c12ac600428f803bff2; no voters:
I20260929 18:33:04.769378 7321 leader_election.cc:288] T 00000000000000000000000000000000 P 4c1d2289be0a4c12ac600428f803bff2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:33:04.769467 7326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c1d2289be0a4c12ac600428f803bff2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:33:04.769387 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:04.748989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34490 (local address 127.28.207.126:43633)
0929 18:33:04.749466 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:04.749616 (+ 150us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:04.749711 (+ 95us) server_negotiation.cc:407] Connection header received
0929 18:33:04.751047 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:04.751084 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:04.751626 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:04.752083 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:04.753207 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:04.754686 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:04.756166 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:04.757175 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:04.757434 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:04.761240 (+ 3806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:04.761247 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:04.761269 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:04.761307 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:04.764283 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:04.765167 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:04.765174 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:04.765180 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:04.765324 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:04.765879 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:04.765889 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:04.765891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:04.766680 (+ 789us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:04.767201 (+ 521us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:04.767672 (+ 471us) server_negotiation.cc:299] Negotiation successful
0929 18:33:04.768114 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":176,"threads_started":1}
I20260929 18:33:04.769711 7326 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c1d2289be0a4c12ac600428f803bff2 [term 1 LEADER]: Becoming Leader. State: Replica: 4c1d2289be0a4c12ac600428f803bff2, State: Running, Role: LEADER
I20260929 18:33:04.770102 7326 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c1d2289be0a4c12ac600428f803bff2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1d2289be0a4c12ac600428f803bff2" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43633 } }
I20260929 18:33:04.771018 7321 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c1d2289be0a4c12ac600428f803bff2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:33:04.773522 7327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c1d2289be0a4c12ac600428f803bff2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c1d2289be0a4c12ac600428f803bff2. Latest consensus state: current_term: 1 leader_uuid: "4c1d2289be0a4c12ac600428f803bff2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1d2289be0a4c12ac600428f803bff2" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43633 } } }
I20260929 18:33:04.773979 7327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c1d2289be0a4c12ac600428f803bff2 [sys.catalog]: This master's current role is: LEADER
I20260929 18:33:04.774626 7328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c1d2289be0a4c12ac600428f803bff2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c1d2289be0a4c12ac600428f803bff2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1d2289be0a4c12ac600428f803bff2" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43633 } } }
I20260929 18:33:04.774947 7328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c1d2289be0a4c12ac600428f803bff2 [sys.catalog]: This master's current role is: LEADER
W20260929 18:33:04.776994 7341 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4c1d2289be0a4c12ac600428f803bff2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:33:04.777105 7341 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:33:04.777248 7342 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:33:04.778064 7342 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:33:04.784138 7342 catalog_manager.cc:1347] Generated new cluster ID: 9901da3423aa4059b265807b2cf9faa8
I20260929 18:33:04.784227 7342 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:33:04.806082 7342 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:33:04.808281 7342 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:33:04.824308 7342 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c1d2289be0a4c12ac600428f803bff2: Generated new TSK 0
I20260929 18:33:04.825214 7342 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.207.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab.
Entry for principal kudu/127.28.207.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab.
I20260929 18:33:04.878007 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.65:0
--local_ip_for_outbound_sockets=127.28.207.65
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:43633
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:33787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:33:05.014129 7347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:33:05.014701 7347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:33:05.015074 7347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:33:05.019898 7347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:33:05.019994 7347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:33:05.020062 7347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.65
I20260929 18:33:05.026182 7347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:33787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:43633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.65
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:33:05.027882 7347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:33:05.029089 7347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:33:05.044214 7353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:05.044266 7347 server_base.cc:1034] running on GCE node
W20260929 18:33:05.044600 7352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:05.044242 7355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:05.045321 7347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:33:05.046373 7347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:33:05.049921 7347 hybrid_clock.cc:648] HybridClock initialized: now 1790706785049812 us; error 88 us; skew 500 ppm
Sep 29 18:33:05 dist-test-slave-qcpv krb5kdc[7240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706785, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:33:05.055505 7347 init.cc:377] Logged in from keytab as kudu/127.28.207.65@KRBTEST.COM (short username kudu)
I20260929 18:33:05.057106 7347 webserver.cc:460] Webserver started at http://127.28.207.65:35771/ using document root <none> and password file <none>
I20260929 18:33:05.058025 7347 fs_manager.cc:359] Metadata directory not provided
I20260929 18:33:05.058169 7347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:33:05.058480 7347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:33:05.061362 7347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/data/instance:
uuid: "59fc421ff8aa44b28deedae788bb10e6"
format_stamp: "Formatted at 2026-09-29 18:33:05 on dist-test-slave-qcpv"
server_key: "f82d59d537fc900b17fc4e65dc690e85"
server_key_iv: "b06cf6df6eb4d4fd167a6813fcb8dc65"
server_key_version: "encryptionkey@0"
I20260929 18:33:05.062318 7347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance:
uuid: "59fc421ff8aa44b28deedae788bb10e6"
format_stamp: "Formatted at 2026-09-29 18:33:05 on dist-test-slave-qcpv"
server_key: "f82d59d537fc900b17fc4e65dc690e85"
server_key_iv: "b06cf6df6eb4d4fd167a6813fcb8dc65"
server_key_version: "encryptionkey@0"
I20260929 18:33:05.068226 7347 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260929 18:33:05.072255 7362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:05.074118 7347 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260929 18:33:05.074254 7347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/wal
uuid: "59fc421ff8aa44b28deedae788bb10e6"
format_stamp: "Formatted at 2026-09-29 18:33:05 on dist-test-slave-qcpv"
server_key: "f82d59d537fc900b17fc4e65dc690e85"
server_key_iv: "b06cf6df6eb4d4fd167a6813fcb8dc65"
server_key_version: "encryptionkey@0"
I20260929 18:33:05.074375 7347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:33:05.093421 7347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:33:05.098434 7347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:33:05.098676 7347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:33:05.099413 7347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:33:05.100340 7347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:33:05.100421 7347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:05.100492 7347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:33:05.100543 7347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:05.113950 7347 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.65:46593
I20260929 18:33:05.113969 7475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.65:46593 every 8 connection(s)
I20260929 18:33:05.115152 7347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
I20260929 18:33:05.118079 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 7347
I20260929 18:33:05.118273 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance
I20260929 18:33:05.118630 29501 external_mini_cluster.cc:1428] Setting key d20773ff1dd6ba213dd6644ff64324af
Sep 29 18:33:05 dist-test-slave-qcpv krb5kdc[7240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706785, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.207.66@KRBTEST.COM; defaulting to no policy
I20260929 18:33:05.133200 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:05.117940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:34513 (local address 127.28.207.126:43633)
0929 18:33:05.118184 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:05.118190 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:05.119178 (+ 988us) server_negotiation.cc:407] Connection header received
0929 18:33:05.120820 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:05.120825 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:05.120893 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:05.121006 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:05.122465 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:05.123117 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:05.124352 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:05.124595 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:05.124721 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:05.128461 (+ 3740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:05.128464 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:05.128467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:05.128496 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:05.130665 (+ 2169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:05.131388 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:05.131393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:05.131398 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:05.131487 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:05.131872 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:05.131875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:05.131885 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:05.132091 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:05.132202 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:05.132858 (+ 656us) server_negotiation.cc:299] Negotiation successful
0929 18:33:05.133006 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260929 18:33:05.134168 7478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:05.118442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43633 (local address 127.28.207.65:34513)
0929 18:33:05.119005 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:05.119064 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:05.120557 (+ 1493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:05.121223 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:05.121230 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:05.121683 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:33:05.122237 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:05.122252 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:05.123255 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:05.123258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:05.124200 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:05.124210 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:05.124777 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:05.124781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:05.124981 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:05.126172 (+ 1191us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:33:05.126190 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:33:05.128226 (+ 2036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:33:05.130819 (+ 2593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:05.130828 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:05.130848 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:05.131276 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:05.131606 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:05.131609 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:05.131611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:05.131765 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:05.132198 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:33:05.132205 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:33:05.132573 (+ 368us) client_negotiation.cc:769] Sending connection context
0929 18:33:05.132803 (+ 230us) client_negotiation.cc:240] Negotiation successful
0929 18:33:05.133157 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":170,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.66@KRBTEST.COM" created.
I20260929 18:33:05.136590 7476 heartbeater.cc:347] Connected to a master server at 127.28.207.126:43633
I20260929 18:33:05.137171 7476 heartbeater.cc:464] Registering TS with master...
I20260929 18:33:05.138790 7476 heartbeater.cc:511] Master 127.28.207.126:43633 requested a full tablet report, sending...
I20260929 18:33:05.141865 7285 ts_manager.cc:194] Registered new tserver with Master: 59fc421ff8aa44b28deedae788bb10e6 (127.28.207.65:46593)
I20260929 18:33:05.143265 7285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.65@KRBTEST.COM'} at 127.28.207.65:34513
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab.
Entry for principal kudu/127.28.207.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab.
I20260929 18:33:05.159796 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.66:0
--local_ip_for_outbound_sockets=127.28.207.66
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:43633
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:33787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:33:05.309243 7481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:33:05.309644 7481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:33:05.309697 7481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:33:05.314783 7481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:33:05.314946 7481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:33:05.315032 7481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.66
I20260929 18:33:05.322487 7481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:33787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:43633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.66
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:33:05.324223 7481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:33:05.325592 7481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:33:05.337852 7486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:05.337846 7487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:05.338057 7481 server_base.cc:1034] running on GCE node
W20260929 18:33:05.337842 7489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:05.339002 7481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:33:05.339848 7481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:33:05.341040 7481 hybrid_clock.cc:648] HybridClock initialized: now 1790706785340999 us; error 68 us; skew 500 ppm
Sep 29 18:33:05 dist-test-slave-qcpv krb5kdc[7240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706785, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:33:05.344419 7481 init.cc:377] Logged in from keytab as kudu/127.28.207.66@KRBTEST.COM (short username kudu)
I20260929 18:33:05.345378 7481 webserver.cc:460] Webserver started at http://127.28.207.66:42955/ using document root <none> and password file <none>
I20260929 18:33:05.346011 7481 fs_manager.cc:359] Metadata directory not provided
I20260929 18:33:05.346077 7481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:33:05.346334 7481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:33:05.348601 7481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/data/instance:
uuid: "9cd646079fd64d0db3bd05dc1cfc35d2"
format_stamp: "Formatted at 2026-09-29 18:33:05 on dist-test-slave-qcpv"
server_key: "3478d20a61a9e77efec5a4576d2fb377"
server_key_iv: "0e35856ec9f127cdc19dde6edb399b9f"
server_key_version: "encryptionkey@0"
I20260929 18:33:05.349263 7481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance:
uuid: "9cd646079fd64d0db3bd05dc1cfc35d2"
format_stamp: "Formatted at 2026-09-29 18:33:05 on dist-test-slave-qcpv"
server_key: "3478d20a61a9e77efec5a4576d2fb377"
server_key_iv: "0e35856ec9f127cdc19dde6edb399b9f"
server_key_version: "encryptionkey@0"
I20260929 18:33:05.353685 7481 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:33:05.357230 7496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:05.359359 7481 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:33:05.359544 7481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/wal
uuid: "9cd646079fd64d0db3bd05dc1cfc35d2"
format_stamp: "Formatted at 2026-09-29 18:33:05 on dist-test-slave-qcpv"
server_key: "3478d20a61a9e77efec5a4576d2fb377"
server_key_iv: "0e35856ec9f127cdc19dde6edb399b9f"
server_key_version: "encryptionkey@0"
I20260929 18:33:05.359685 7481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:33:05.389093 7481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:33:05.393187 7481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:33:05.393441 7481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:33:05.394095 7481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:33:05.395062 7481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:33:05.395136 7481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:05.395197 7481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:33:05.395248 7481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:05.406294 7481 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.66:44651
I20260929 18:33:05.406337 7609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.66:44651 every 8 connection(s)
I20260929 18:33:05.407380 7481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
I20260929 18:33:05.410293 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 7481
I20260929 18:33:05.410590 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance
I20260929 18:33:05.411062 29501 external_mini_cluster.cc:1428] Setting key 1e52f8204b83cd54d4ef8e7d4705995d
Sep 29 18:33:05 dist-test-slave-qcpv krb5kdc[7240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706785, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.207.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.67@KRBTEST.COM" created.
I20260929 18:33:05.429292 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:05.410513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:37343 (local address 127.28.207.126:43633)
0929 18:33:05.410712 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:05.410717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:05.411944 (+ 1227us) server_negotiation.cc:407] Connection header received
0929 18:33:05.414167 (+ 2223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:05.414170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:05.414227 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:05.414339 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:05.416450 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:05.417321 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:05.419485 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:05.419742 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:05.419867 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:05.423721 (+ 3854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:05.423725 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:05.423727 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:05.423757 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:05.425561 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:05.426383 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:05.426391 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:05.426393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:05.426674 (+ 281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:05.427462 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:05.427465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:05.427466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:05.427682 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:05.427810 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:05.428870 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0929 18:33:05.429054 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 18:33:05.430621 7612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:05.410957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43633 (local address 127.28.207.66:37343)
0929 18:33:05.411680 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:05.411748 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:05.413819 (+ 2071us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:05.414644 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:05.414659 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:05.415385 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:33:05.416234 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:05.416252 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:05.417519 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:05.417523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:05.419283 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:05.419301 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:05.419982 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:05.419990 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:05.420287 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:05.421151 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:33:05.421182 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:33:05.423464 (+ 2282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:33:05.425847 (+ 2383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:05.425854 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:05.425873 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:05.426224 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:05.426999 (+ 775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:05.427003 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:05.427005 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:05.427285 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:05.427850 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:33:05.427858 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:33:05.428308 (+ 450us) client_negotiation.cc:769] Sending connection context
0929 18:33:05.428758 (+ 450us) client_negotiation.cc:240] Negotiation successful
0929 18:33:05.429315 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":184,"threads_started":1}
I20260929 18:33:05.432492 7610 heartbeater.cc:347] Connected to a master server at 127.28.207.126:43633
I20260929 18:33:05.432811 7610 heartbeater.cc:464] Registering TS with master...
I20260929 18:33:05.433423 7610 heartbeater.cc:511] Master 127.28.207.126:43633 requested a full tablet report, sending...
I20260929 18:33:05.434762 7285 ts_manager.cc:194] Registered new tserver with Master: 9cd646079fd64d0db3bd05dc1cfc35d2 (127.28.207.66:44651)
I20260929 18:33:05.435544 7285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.66@KRBTEST.COM'} at 127.28.207.66:37343
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab.
Entry for principal kudu/127.28.207.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab.
I20260929 18:33:05.451416 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.67:0
--local_ip_for_outbound_sockets=127.28.207.67
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:43633
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:33787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:33:05.584386 7615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:33:05.584769 7615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:33:05.584864 7615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:33:05.589498 7615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:33:05.589637 7615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:33:05.589742 7615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.67
I20260929 18:33:05.595865 7615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:33787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:43633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.67
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:33:05.597136 7615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:33:05.598311 7615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:33:05.609838 7620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:05.609804 7621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:05.610141 7615 server_base.cc:1034] running on GCE node
W20260929 18:33:05.610335 7623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:05.610980 7615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:33:05.611817 7615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:33:05.613119 7615 hybrid_clock.cc:648] HybridClock initialized: now 1790706785613092 us; error 47 us; skew 500 ppm
Sep 29 18:33:05 dist-test-slave-qcpv krb5kdc[7240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706785, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:33:05.617138 7615 init.cc:377] Logged in from keytab as kudu/127.28.207.67@KRBTEST.COM (short username kudu)
I20260929 18:33:05.618610 7615 webserver.cc:460] Webserver started at http://127.28.207.67:35559/ using document root <none> and password file <none>
I20260929 18:33:05.619503 7615 fs_manager.cc:359] Metadata directory not provided
I20260929 18:33:05.619592 7615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:33:05.619882 7615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:33:05.622745 7615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/data/instance:
uuid: "21ad1ac2fcb84c5c8dc8b02746599ae7"
format_stamp: "Formatted at 2026-09-29 18:33:05 on dist-test-slave-qcpv"
server_key: "7407c7637e16ec39e65e100fcf950503"
server_key_iv: "8f34779f19fbdc89d21ba4d576a9125f"
server_key_version: "encryptionkey@0"
I20260929 18:33:05.623534 7615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance:
uuid: "21ad1ac2fcb84c5c8dc8b02746599ae7"
format_stamp: "Formatted at 2026-09-29 18:33:05 on dist-test-slave-qcpv"
server_key: "7407c7637e16ec39e65e100fcf950503"
server_key_iv: "8f34779f19fbdc89d21ba4d576a9125f"
server_key_version: "encryptionkey@0"
I20260929 18:33:05.629099 7615 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 18:33:05.634410 7630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:05.636574 7615 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:33:05.636830 7615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/wal
uuid: "21ad1ac2fcb84c5c8dc8b02746599ae7"
format_stamp: "Formatted at 2026-09-29 18:33:05 on dist-test-slave-qcpv"
server_key: "7407c7637e16ec39e65e100fcf950503"
server_key_iv: "8f34779f19fbdc89d21ba4d576a9125f"
server_key_version: "encryptionkey@0"
I20260929 18:33:05.637082 7615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:33:05.666615 7615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:33:05.671902 7615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:33:05.672240 7615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:33:05.673142 7615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:33:05.674315 7615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:33:05.674394 7615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:05.674479 7615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:33:05.674525 7615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:05.688463 7615 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.67:46303
I20260929 18:33:05.688503 7743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.67:46303 every 8 connection(s)
I20260929 18:33:05.690011 7615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
I20260929 18:33:05.692936 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 7615
I20260929 18:33:05.693110 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance
I20260929 18:33:05.693647 29501 external_mini_cluster.cc:1428] Setting key 5e2ded49543cc613cc743a25e5bf2f29
Sep 29 18:33:05 dist-test-slave-qcpv krb5kdc[7240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706785, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:33:05.709360 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:05.692638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:60875 (local address 127.28.207.126:43633)
0929 18:33:05.692857 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:05.692863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:05.693812 (+ 949us) server_negotiation.cc:407] Connection header received
0929 18:33:05.695700 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:05.695703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:05.695755 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:05.695832 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:05.697351 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:05.698091 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:05.699527 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:05.699772 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:05.699905 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:05.703300 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:05.703304 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:05.703306 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:05.703369 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:05.705132 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:05.706609 (+ 1477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:05.706618 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:05.706619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:05.706755 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:05.707529 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:05.707533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:05.707535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:05.707908 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:05.708088 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:05.708805 (+ 717us) server_negotiation.cc:299] Negotiation successful
0929 18:33:05.709190 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 18:33:05.711150 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:05.693024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43633 (local address 127.28.207.67:60875)
0929 18:33:05.693638 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:05.693694 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:05.695434 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:05.696037 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:05.696044 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:05.696430 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:33:05.697162 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:05.697183 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:05.698313 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:05.698316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:05.699355 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:05.699366 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:05.699954 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:05.699957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:05.700133 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:05.700878 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:33:05.700900 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:33:05.703029 (+ 2129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:33:05.705302 (+ 2273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:05.705311 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:05.705336 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:05.706395 (+ 1059us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:05.707169 (+ 774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:05.707172 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:05.707174 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:05.707391 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:05.708086 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:33:05.708096 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:33:05.708497 (+ 401us) client_negotiation.cc:769] Sending connection context
0929 18:33:05.708745 (+ 248us) client_negotiation.cc:240] Negotiation successful
0929 18:33:05.709498 (+ 753us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":203,"threads_started":1}
I20260929 18:33:05.712839 7744 heartbeater.cc:347] Connected to a master server at 127.28.207.126:43633
I20260929 18:33:05.713186 7744 heartbeater.cc:464] Registering TS with master...
I20260929 18:33:05.714421 7744 heartbeater.cc:511] Master 127.28.207.126:43633 requested a full tablet report, sending...
I20260929 18:33:05.716109 7285 ts_manager.cc:194] Registered new tserver with Master: 21ad1ac2fcb84c5c8dc8b02746599ae7 (127.28.207.67:46303)
I20260929 18:33:05.717057 7285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.67@KRBTEST.COM'} at 127.28.207.67:60875
I20260929 18:33:05.722128 29501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:33:05.734279 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:05.724647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38458 (local address 127.28.207.126:43633)
0929 18:33:05.724833 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:05.724842 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:05.725099 (+ 257us) server_negotiation.cc:407] Connection header received
0929 18:33:05.725278 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:05.725281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:05.725331 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:05.725454 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:05.726732 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:05.727420 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:05.728408 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:05.728630 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:05.728728 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:05.730469 (+ 1741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:05.730472 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:05.730474 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:05.730505 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:05.732161 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:05.732703 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:05.732706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:05.732707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:05.732784 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:05.733175 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:05.733231 (+ 56us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:05.733236 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:05.733571 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:05.733723 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:05.733986 (+ 263us) server_negotiation.cc:299] Negotiation successful
0929 18:33:05.734108 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:33:05.739936 7285 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38458:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:33:05.744910 7285 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:33:05.765859 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:05.759902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58930 (local address 127.28.207.65:46593)
0929 18:33:05.760342 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:05.760348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:05.760371 (+ 23us) server_negotiation.cc:407] Connection header received
0929 18:33:05.760555 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:05.760561 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:05.760737 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:05.760883 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:33:05.761198 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:05.762011 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:05.763929 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:05.765019 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:05.765183 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:05.765384 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:05.765533 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 18:33:05.765656 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":154,"threads_started":1}
I20260929 18:33:05.766151 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:05.758718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57098 (local address 127.28.207.66:44651)
0929 18:33:05.759276 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:05.759441 (+ 165us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:05.759799 (+ 358us) server_negotiation.cc:407] Connection header received
0929 18:33:05.760082 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:05.760089 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:05.760359 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:05.760508 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:33:05.760980 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:05.762280 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:05.764281 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:05.765077 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:05.765208 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:05.765386 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:05.765881 (+ 495us) server_negotiation.cc:299] Negotiation successful
0929 18:33:05.766002 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":209,"threads_started":1}
I20260929 18:33:05.766558 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:05.758713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49736 (local address 127.28.207.67:46303)
0929 18:33:05.760001 (+ 1288us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:05.760015 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:05.760047 (+ 32us) server_negotiation.cc:407] Connection header received
0929 18:33:05.760157 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:05.760163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:05.760411 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:05.760601 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:33:05.761023 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:05.762523 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:05.764119 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:05.765077 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:05.765208 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:05.765386 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:05.766271 (+ 885us) server_negotiation.cc:299] Negotiation successful
0929 18:33:05.766387 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1162,"thread_start_us":149,"threads_started":1}
I20260929 18:33:05.766911 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:05.758406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:46303 (local address 127.0.0.1:49736)
0929 18:33:05.759396 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:05.759468 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:05.759647 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:05.760622 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:05.760635 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:05.760662 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:33:05.760835 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:05.760842 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:05.762697 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:05.762700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:05.763978 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:05.763988 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:05.765464 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:05.765469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:05.765632 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:05.765719 (+ 87us) client_negotiation.cc:769] Sending connection context
0929 18:33:05.765809 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 18:33:05.765920 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"mutex_wait_us":48,"thread_start_us":134,"threads_started":1}
I20260929 18:33:05.766913 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:05.759406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:46593 (local address 127.0.0.1:58930)
0929 18:33:05.760278 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:05.760295 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:05.760389 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:05.760879 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:05.760883 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:05.760903 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:33:05.761052 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:05.761068 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:05.762157 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:05.762161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:05.763780 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:05.763791 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:05.765182 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:05.765186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:05.765316 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:05.765359 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 18:33:05.765497 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 18:33:05.765661 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"mutex_wait_us":18,"thread_start_us":113,"threads_started":1}
I20260929 18:33:05.766942 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:05.758419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:44651 (local address 127.0.0.1:57098)
0929 18:33:05.759660 (+ 1241us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:05.759682 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:05.759902 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:05.760743 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:05.760746 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:05.760761 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:33:05.760835 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:05.760842 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:05.762469 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:05.762473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:05.764059 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:05.764068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:05.765464 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:05.765469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:05.765651 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:05.765725 (+ 74us) client_negotiation.cc:769] Sending connection context
0929 18:33:05.766439 (+ 714us) client_negotiation.cc:240] Negotiation successful
0929 18:33:05.766533 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1149,"mutex_wait_us":283,"thread_start_us":122,"threads_started":1}
I20260929 18:33:05.771035 7678 tablet_service.cc:1467] Processing CreateTablet for tablet 36a132f8994e469f8b9889c0c03554da (DEFAULT_TABLE table=test-table [id=b604dbbe860a4353b5198180fca6752c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:33:05.771035 7544 tablet_service.cc:1467] Processing CreateTablet for tablet 36a132f8994e469f8b9889c0c03554da (DEFAULT_TABLE table=test-table [id=b604dbbe860a4353b5198180fca6752c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:33:05.771301 7408 tablet_service.cc:1467] Processing CreateTablet for tablet 36a132f8994e469f8b9889c0c03554da (DEFAULT_TABLE table=test-table [id=b604dbbe860a4353b5198180fca6752c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:33:05.772289 7544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36a132f8994e469f8b9889c0c03554da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:33:05.772290 7678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36a132f8994e469f8b9889c0c03554da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:33:05.772866 7408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36a132f8994e469f8b9889c0c03554da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:33:05.784026 7763 tablet_bootstrap.cc:492] T 36a132f8994e469f8b9889c0c03554da P 59fc421ff8aa44b28deedae788bb10e6: Bootstrap starting.
I20260929 18:33:05.784272 7761 tablet_bootstrap.cc:492] T 36a132f8994e469f8b9889c0c03554da P 21ad1ac2fcb84c5c8dc8b02746599ae7: Bootstrap starting.
I20260929 18:33:05.786247 7763 tablet_bootstrap.cc:654] T 36a132f8994e469f8b9889c0c03554da P 59fc421ff8aa44b28deedae788bb10e6: Neither blocks nor log segments found. Creating new log.
I20260929 18:33:05.787130 7763 log.cc:824] T 36a132f8994e469f8b9889c0c03554da P 59fc421ff8aa44b28deedae788bb10e6: Log is configured to *not* fsync() on all Append() calls
I20260929 18:33:05.787182 7761 tablet_bootstrap.cc:654] T 36a132f8994e469f8b9889c0c03554da P 21ad1ac2fcb84c5c8dc8b02746599ae7: Neither blocks nor log segments found. Creating new log.
I20260929 18:33:05.788090 7761 log.cc:824] T 36a132f8994e469f8b9889c0c03554da P 21ad1ac2fcb84c5c8dc8b02746599ae7: Log is configured to *not* fsync() on all Append() calls
I20260929 18:33:05.788815 7762 tablet_bootstrap.cc:492] T 36a132f8994e469f8b9889c0c03554da P 9cd646079fd64d0db3bd05dc1cfc35d2: Bootstrap starting.
I20260929 18:33:05.789355 7763 tablet_bootstrap.cc:492] T 36a132f8994e469f8b9889c0c03554da P 59fc421ff8aa44b28deedae788bb10e6: No bootstrap required, opened a new log
I20260929 18:33:05.789613 7763 ts_tablet_manager.cc:1397] T 36a132f8994e469f8b9889c0c03554da P 59fc421ff8aa44b28deedae788bb10e6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:33:05.791491 7761 tablet_bootstrap.cc:492] T 36a132f8994e469f8b9889c0c03554da P 21ad1ac2fcb84c5c8dc8b02746599ae7: No bootstrap required, opened a new log
I20260929 18:33:05.791745 7761 ts_tablet_manager.cc:1397] T 36a132f8994e469f8b9889c0c03554da P 21ad1ac2fcb84c5c8dc8b02746599ae7: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 18:33:05.792564 7762 tablet_bootstrap.cc:654] T 36a132f8994e469f8b9889c0c03554da P 9cd646079fd64d0db3bd05dc1cfc35d2: Neither blocks nor log segments found. Creating new log.
I20260929 18:33:05.793707 7762 log.cc:824] T 36a132f8994e469f8b9889c0c03554da P 9cd646079fd64d0db3bd05dc1cfc35d2: Log is configured to *not* fsync() on all Append() calls
I20260929 18:33:05.795120 7763 raft_consensus.cc:348] T 36a132f8994e469f8b9889c0c03554da P 59fc421ff8aa44b28deedae788bb10e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ad1ac2fcb84c5c8dc8b02746599ae7" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 46303 } } peers { permanent_uuid: "9cd646079fd64d0db3bd05dc1cfc35d2" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 44651 } } peers { permanent_uuid: "59fc421ff8aa44b28deedae788bb10e6" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46593 } }
I20260929 18:33:05.795450 7763 raft_consensus.cc:374] T 36a132f8994e469f8b9889c0c03554da P 59fc421ff8aa44b28deedae788bb10e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:33:05.795506 7763 raft_consensus.cc:729] T 36a132f8994e469f8b9889c0c03554da P 59fc421ff8aa44b28deedae788bb10e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59fc421ff8aa44b28deedae788bb10e6, State: Initialized, Role: FOLLOWER
I20260929 18:33:05.795989 7763 consensus_queue.cc:260] T 36a132f8994e469f8b9889c0c03554da P 59fc421ff8aa44b28deedae788bb10e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ad1ac2fcb84c5c8dc8b02746599ae7" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 46303 } } peers { permanent_uuid: "9cd646079fd64d0db3bd05dc1cfc35d2" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 44651 } } peers { permanent_uuid: "59fc421ff8aa44b28deedae788bb10e6" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46593 } }
I20260929 18:33:05.796205 7761 raft_consensus.cc:348] T 36a132f8994e469f8b9889c0c03554da P 21ad1ac2fcb84c5c8dc8b02746599ae7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ad1ac2fcb84c5c8dc8b02746599ae7" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 46303 } } peers { permanent_uuid: "9cd646079fd64d0db3bd05dc1cfc35d2" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 44651 } } peers { permanent_uuid: "59fc421ff8aa44b28deedae788bb10e6" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46593 } }
I20260929 18:33:05.796525 7761 raft_consensus.cc:374] T 36a132f8994e469f8b9889c0c03554da P 21ad1ac2fcb84c5c8dc8b02746599ae7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:33:05.796588 7761 raft_consensus.cc:729] T 36a132f8994e469f8b9889c0c03554da P 21ad1ac2fcb84c5c8dc8b02746599ae7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21ad1ac2fcb84c5c8dc8b02746599ae7, State: Initialized, Role: FOLLOWER
I20260929 18:33:05.796662 7762 tablet_bootstrap.cc:492] T 36a132f8994e469f8b9889c0c03554da P 9cd646079fd64d0db3bd05dc1cfc35d2: No bootstrap required, opened a new log
I20260929 18:33:05.796903 7762 ts_tablet_manager.cc:1397] T 36a132f8994e469f8b9889c0c03554da P 9cd646079fd64d0db3bd05dc1cfc35d2: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 18:33:05.796998 7476 heartbeater.cc:503] Master 127.28.207.126:43633 was elected leader, sending a full tablet report...
I20260929 18:33:05.797178 7761 consensus_queue.cc:260] T 36a132f8994e469f8b9889c0c03554da P 21ad1ac2fcb84c5c8dc8b02746599ae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ad1ac2fcb84c5c8dc8b02746599ae7" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 46303 } } peers { permanent_uuid: "9cd646079fd64d0db3bd05dc1cfc35d2" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 44651 } } peers { permanent_uuid: "59fc421ff8aa44b28deedae788bb10e6" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46593 } }
I20260929 18:33:05.797333 7763 ts_tablet_manager.cc:1428] T 36a132f8994e469f8b9889c0c03554da P 59fc421ff8aa44b28deedae788bb10e6: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260929 18:33:05.798062 7744 heartbeater.cc:503] Master 127.28.207.126:43633 was elected leader, sending a full tablet report...
I20260929 18:33:05.798503 7761 ts_tablet_manager.cc:1428] T 36a132f8994e469f8b9889c0c03554da P 21ad1ac2fcb84c5c8dc8b02746599ae7: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 18:33:05.801827 7762 raft_consensus.cc:348] T 36a132f8994e469f8b9889c0c03554da P 9cd646079fd64d0db3bd05dc1cfc35d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ad1ac2fcb84c5c8dc8b02746599ae7" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 46303 } } peers { permanent_uuid: "9cd646079fd64d0db3bd05dc1cfc35d2" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 44651 } } peers { permanent_uuid: "59fc421ff8aa44b28deedae788bb10e6" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46593 } }
I20260929 18:33:05.802157 7762 raft_consensus.cc:374] T 36a132f8994e469f8b9889c0c03554da P 9cd646079fd64d0db3bd05dc1cfc35d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:33:05.802227 7762 raft_consensus.cc:729] T 36a132f8994e469f8b9889c0c03554da P 9cd646079fd64d0db3bd05dc1cfc35d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cd646079fd64d0db3bd05dc1cfc35d2, State: Initialized, Role: FOLLOWER
I20260929 18:33:05.802832 7762 consensus_queue.cc:260] T 36a132f8994e469f8b9889c0c03554da P 9cd646079fd64d0db3bd05dc1cfc35d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ad1ac2fcb84c5c8dc8b02746599ae7" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 46303 } } peers { permanent_uuid: "9cd646079fd64d0db3bd05dc1cfc35d2" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 44651 } } peers { permanent_uuid: "59fc421ff8aa44b28deedae788bb10e6" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46593 } }
I20260929 18:33:05.803781 7610 heartbeater.cc:503] Master 127.28.207.126:43633 was elected leader, sending a full tablet report...
I20260929 18:33:05.804209 7762 ts_tablet_manager.cc:1428] T 36a132f8994e469f8b9889c0c03554da P 9cd646079fd64d0db3bd05dc1cfc35d2: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
W20260929 18:33:05.867100 7477 tablet.cc:2369] T 36a132f8994e469f8b9889c0c03554da P 59fc421ff8aa44b28deedae788bb10e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:33:05.909037 7611 tablet.cc:2369] T 36a132f8994e469f8b9889c0c03554da P 9cd646079fd64d0db3bd05dc1cfc35d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:33:05.928609 7769 raft_consensus.cc:482] T 36a132f8994e469f8b9889c0c03554da P 9cd646079fd64d0db3bd05dc1cfc35d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:33:05.928877 7769 raft_consensus.cc:504] T 36a132f8994e469f8b9889c0c03554da P 9cd646079fd64d0db3bd05dc1cfc35d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ad1ac2fcb84c5c8dc8b02746599ae7" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 46303 } } peers { permanent_uuid: "9cd646079fd64d0db3bd05dc1cfc35d2" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 44651 } } peers { permanent_uuid: "59fc421ff8aa44b28deedae788bb10e6" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46593 } }
I20260929 18:33:05.930393 7769 leader_election.cc:288] T 36a132f8994e469f8b9889c0c03554da P 9cd646079fd64d0db3bd05dc1cfc35d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21ad1ac2fcb84c5c8dc8b02746599ae7 (127.28.207.67:46303), 59fc421ff8aa44b28deedae788bb10e6 (127.28.207.65:46593)
I20260929 18:33:05.934841 7612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:05.930930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:46303 (local address 127.28.207.66:39649)
0929 18:33:05.931099 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:05.931120 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:05.931322 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:05.931695 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:05.931698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:05.931715 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:33:05.931867 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:05.931873 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:05.932684 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:05.932691 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:05.933578 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:05.933587 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:05.934310 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:05.934319 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:05.934450 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:05.934470 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 18:33:05.934567 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 18:33:05.934664 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":4}
I20260929 18:33:05.934882 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:05.930907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:39649 (local address 127.28.207.67:46303)
0929 18:33:05.931079 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:05.931084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:05.931249 (+ 165us) server_negotiation.cc:407] Connection header received
0929 18:33:05.931522 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:05.931526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:05.931582 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:05.931681 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:33:05.931989 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:05.932560 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:05.933720 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:05.934169 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:05.934319 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:05.934389 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:05.934584 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 18:33:05.934668 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 18:33:05.935515 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:05.930930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:46593 (local address 127.28.207.66:45683)
0929 18:33:05.931503 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:05.931517 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:05.931680 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:05.931980 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:05.931983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:05.932000 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:33:05.932149 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:05.932155 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:05.933035 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:05.933040 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:05.933964 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:05.933973 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:05.934923 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:05.934926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:05.935056 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:05.935097 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 18:33:05.935197 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 18:33:05.935306 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":179,"thread_start_us":113,"threads_started":1}
I20260929 18:33:05.935535 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:05.931013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:45683 (local address 127.28.207.65:46593)
0929 18:33:05.931157 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:05.931161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:05.931622 (+ 461us) server_negotiation.cc:407] Connection header received
0929 18:33:05.931817 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:05.931820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:05.931877 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:05.931957 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:33:05.932260 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:05.932860 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:05.934112 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:05.934566 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:05.934744 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:05.934808 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:05.935249 (+ 441us) server_negotiation.cc:299] Negotiation successful
0929 18:33:05.935380 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:33:05.935870 7698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36a132f8994e469f8b9889c0c03554da" candidate_uuid: "9cd646079fd64d0db3bd05dc1cfc35d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21ad1ac2fcb84c5c8dc8b02746599ae7" is_pre_election: true
I20260929 18:33:05.936321 7698 raft_consensus.cc:2454] T 36a132f8994e469f8b9889c0c03554da P 21ad1ac2fcb84c5c8dc8b02746599ae7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cd646079fd64d0db3bd05dc1cfc35d2 in term 0.
I20260929 18:33:05.936299 7430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36a132f8994e469f8b9889c0c03554da" candidate_uuid: "9cd646079fd64d0db3bd05dc1cfc35d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59fc421ff8aa44b28deedae788bb10e6" is_pre_election: true
I20260929 18:33:05.936551 7430 raft_consensus.cc:2454] T 36a132f8994e469f8b9889c0c03554da P 59fc421ff8aa44b28deedae788bb10e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cd646079fd64d0db3bd05dc1cfc35d2 in term 0.
I20260929 18:33:05.937011 7500 leader_election.cc:302] T 36a132f8994e469f8b9889c0c03554da P 9cd646079fd64d0db3bd05dc1cfc35d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21ad1ac2fcb84c5c8dc8b02746599ae7, 9cd646079fd64d0db3bd05dc1cfc35d2; no voters:
I20260929 18:33:05.937363 7769 raft_consensus.cc:2781] T 36a132f8994e469f8b9889c0c03554da P 9cd646079fd64d0db3bd05dc1cfc35d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:33:05.937491 7769 raft_consensus.cc:482] T 36a132f8994e469f8b9889c0c03554da P 9cd646079fd64d0db3bd05dc1cfc35d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:33:05.937638 7769 raft_consensus.cc:3037] T 36a132f8994e469f8b9889c0c03554da P 9cd646079fd64d0db3bd05dc1cfc35d2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:33:05.938949 7769 raft_consensus.cc:504] T 36a132f8994e469f8b9889c0c03554da P 9cd646079fd64d0db3bd05dc1cfc35d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ad1ac2fcb84c5c8dc8b02746599ae7" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 46303 } } peers { permanent_uuid: "9cd646079fd64d0db3bd05dc1cfc35d2" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 44651 } } peers { permanent_uuid: "59fc421ff8aa44b28deedae788bb10e6" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46593 } }
I20260929 18:33:05.939571 7769 leader_election.cc:288] T 36a132f8994e469f8b9889c0c03554da P 9cd646079fd64d0db3bd05dc1cfc35d2 [CANDIDATE]: Term 1 election: Requested vote from peers 21ad1ac2fcb84c5c8dc8b02746599ae7 (127.28.207.67:46303), 59fc421ff8aa44b28deedae788bb10e6 (127.28.207.65:46593)
I20260929 18:33:05.940251 7430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36a132f8994e469f8b9889c0c03554da" candidate_uuid: "9cd646079fd64d0db3bd05dc1cfc35d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59fc421ff8aa44b28deedae788bb10e6"
I20260929 18:33:05.940294 7698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36a132f8994e469f8b9889c0c03554da" candidate_uuid: "9cd646079fd64d0db3bd05dc1cfc35d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21ad1ac2fcb84c5c8dc8b02746599ae7"
I20260929 18:33:05.940436 7430 raft_consensus.cc:3037] T 36a132f8994e469f8b9889c0c03554da P 59fc421ff8aa44b28deedae788bb10e6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:33:05.940516 7698 raft_consensus.cc:3037] T 36a132f8994e469f8b9889c0c03554da P 21ad1ac2fcb84c5c8dc8b02746599ae7 [term 0 FOLLOWER]: Advancing to term 1
W20260929 18:33:05.941421 7745 tablet.cc:2369] T 36a132f8994e469f8b9889c0c03554da P 21ad1ac2fcb84c5c8dc8b02746599ae7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:33:05.941886 7430 raft_consensus.cc:2454] T 36a132f8994e469f8b9889c0c03554da P 59fc421ff8aa44b28deedae788bb10e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cd646079fd64d0db3bd05dc1cfc35d2 in term 1.
I20260929 18:33:05.942246 7498 leader_election.cc:302] T 36a132f8994e469f8b9889c0c03554da P 9cd646079fd64d0db3bd05dc1cfc35d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59fc421ff8aa44b28deedae788bb10e6, 9cd646079fd64d0db3bd05dc1cfc35d2; no voters:
I20260929 18:33:05.942443 7769 raft_consensus.cc:2781] T 36a132f8994e469f8b9889c0c03554da P 9cd646079fd64d0db3bd05dc1cfc35d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:33:05.942456 7698 raft_consensus.cc:2454] T 36a132f8994e469f8b9889c0c03554da P 21ad1ac2fcb84c5c8dc8b02746599ae7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cd646079fd64d0db3bd05dc1cfc35d2 in term 1.
I20260929 18:33:05.942734 7769 raft_consensus.cc:686] T 36a132f8994e469f8b9889c0c03554da P 9cd646079fd64d0db3bd05dc1cfc35d2 [term 1 LEADER]: Becoming Leader. State: Replica: 9cd646079fd64d0db3bd05dc1cfc35d2, State: Running, Role: LEADER
I20260929 18:33:05.943185 7769 consensus_queue.cc:237] T 36a132f8994e469f8b9889c0c03554da P 9cd646079fd64d0db3bd05dc1cfc35d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ad1ac2fcb84c5c8dc8b02746599ae7" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 46303 } } peers { permanent_uuid: "9cd646079fd64d0db3bd05dc1cfc35d2" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 44651 } } peers { permanent_uuid: "59fc421ff8aa44b28deedae788bb10e6" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46593 } }
I20260929 18:33:05.946485 7285 catalog_manager.cc:5495] T 36a132f8994e469f8b9889c0c03554da P 9cd646079fd64d0db3bd05dc1cfc35d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cd646079fd64d0db3bd05dc1cfc35d2 (127.28.207.66). New cstate: current_term: 1 leader_uuid: "9cd646079fd64d0db3bd05dc1cfc35d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21ad1ac2fcb84c5c8dc8b02746599ae7" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 46303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cd646079fd64d0db3bd05dc1cfc35d2" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 44651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59fc421ff8aa44b28deedae788bb10e6" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46593 } health_report { overall_health: UNKNOWN } } }
I20260929 18:33:05.963208 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:05.959591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57104 (local address 127.28.207.66:44651)
0929 18:33:05.959751 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:05.959762 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:05.959775 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:33:05.959852 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:05.959854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:05.959907 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:05.960023 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:33:05.960257 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:05.960758 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:05.961621 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:05.961870 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:05.961985 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:05.962305 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:33:05.962729 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:33:05.962922 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:05.962960 (+ 38us) server_negotiation.cc:299] Negotiation successful
0929 18:33:05.963028 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 18:33:05.971014 7430 raft_consensus.cc:1261] T 36a132f8994e469f8b9889c0c03554da P 59fc421ff8aa44b28deedae788bb10e6 [term 1 FOLLOWER]: Refusing update from remote peer 9cd646079fd64d0db3bd05dc1cfc35d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:33:05.971027 7698 raft_consensus.cc:1261] T 36a132f8994e469f8b9889c0c03554da P 21ad1ac2fcb84c5c8dc8b02746599ae7 [term 1 FOLLOWER]: Refusing update from remote peer 9cd646079fd64d0db3bd05dc1cfc35d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:33:05.972007 7769 consensus_queue.cc:1035] T 36a132f8994e469f8b9889c0c03554da P 9cd646079fd64d0db3bd05dc1cfc35d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59fc421ff8aa44b28deedae788bb10e6" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:33:05.972277 7771 consensus_queue.cc:1035] T 36a132f8994e469f8b9889c0c03554da P 9cd646079fd64d0db3bd05dc1cfc35d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21ad1ac2fcb84c5c8dc8b02746599ae7" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:33:05.985590 7775 mvcc.cc:205] Tried to move back new op lower bound from 7334734995327778816 to 7334734995226189824. Current Snapshot: MvccSnapshot[applied={T|T < 7334734995327778816}]
I20260929 18:33:05.994620 7285 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:33:05.994915 7285 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:33:05.998232 7285 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c1d2289be0a4c12ac600428f803bff2: Sending DeleteTablet for 3 replicas of tablet 36a132f8994e469f8b9889c0c03554da
I20260929 18:33:05.999336 7544 tablet_service.cc:1514] Processing DeleteTablet for tablet 36a132f8994e469f8b9889c0c03554da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:33:05 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:57098
I20260929 18:33:05.999492 7408 tablet_service.cc:1514] Processing DeleteTablet for tablet 36a132f8994e469f8b9889c0c03554da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:33:05 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:58930
I20260929 18:33:05.999439 7678 tablet_service.cc:1514] Processing DeleteTablet for tablet 36a132f8994e469f8b9889c0c03554da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:33:05 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:49736
I20260929 18:33:06.000352 7783 tablet_replica.cc:330] T 36a132f8994e469f8b9889c0c03554da P 9cd646079fd64d0db3bd05dc1cfc35d2: stopping tablet replica
I20260929 18:33:06.000909 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 7347
I20260929 18:33:06.000898 7783 raft_consensus.cc:2229] T 36a132f8994e469f8b9889c0c03554da P 9cd646079fd64d0db3bd05dc1cfc35d2 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:33:06.002023 7783 raft_consensus.cc:2258] T 36a132f8994e469f8b9889c0c03554da P 9cd646079fd64d0db3bd05dc1cfc35d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:33:06.000737 7784 tablet_replica.cc:330] T 36a132f8994e469f8b9889c0c03554da P 21ad1ac2fcb84c5c8dc8b02746599ae7: stopping tablet replica
I20260929 18:33:06.004200 7784 raft_consensus.cc:2229] T 36a132f8994e469f8b9889c0c03554da P 21ad1ac2fcb84c5c8dc8b02746599ae7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:33:06.004241 7783 ts_tablet_manager.cc:1905] T 36a132f8994e469f8b9889c0c03554da P 9cd646079fd64d0db3bd05dc1cfc35d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:33:06.004642 7784 raft_consensus.cc:2258] T 36a132f8994e469f8b9889c0c03554da P 21ad1ac2fcb84c5c8dc8b02746599ae7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:33:06.006915 7784 ts_tablet_manager.cc:1905] T 36a132f8994e469f8b9889c0c03554da P 21ad1ac2fcb84c5c8dc8b02746599ae7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:33:06.008540 7783 ts_tablet_manager.cc:1918] T 36a132f8994e469f8b9889c0c03554da P 9cd646079fd64d0db3bd05dc1cfc35d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:33:06.008684 7783 log.cc:1196] T 36a132f8994e469f8b9889c0c03554da P 9cd646079fd64d0db3bd05dc1cfc35d2: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-1/wal/wals/36a132f8994e469f8b9889c0c03554da
I20260929 18:33:06.009047 7783 ts_tablet_manager.cc:1939] T 36a132f8994e469f8b9889c0c03554da P 9cd646079fd64d0db3bd05dc1cfc35d2: Deleting consensus metadata
I20260929 18:33:06.010380 7784 ts_tablet_manager.cc:1918] T 36a132f8994e469f8b9889c0c03554da P 21ad1ac2fcb84c5c8dc8b02746599ae7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:33:06.010502 7784 log.cc:1196] T 36a132f8994e469f8b9889c0c03554da P 21ad1ac2fcb84c5c8dc8b02746599ae7: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790706718260899-29501-0/minicluster-data/ts-2/wal/wals/36a132f8994e469f8b9889c0c03554da
I20260929 18:33:06.010888 7784 ts_tablet_manager.cc:1939] T 36a132f8994e469f8b9889c0c03554da P 21ad1ac2fcb84c5c8dc8b02746599ae7: Deleting consensus metadata
I20260929 18:33:06.011330 7270 catalog_manager.cc:4841] TS 9cd646079fd64d0db3bd05dc1cfc35d2 (127.28.207.66:44651): tablet 36a132f8994e469f8b9889c0c03554da (table test-table [id=b604dbbe860a4353b5198180fca6752c]) successfully deleted
I20260929 18:33:06.012399 7272 catalog_manager.cc:4841] TS 21ad1ac2fcb84c5c8dc8b02746599ae7 (127.28.207.67:46303): tablet 36a132f8994e469f8b9889c0c03554da (table test-table [id=b604dbbe860a4353b5198180fca6752c]) successfully deleted
W20260929 18:33:06.017587 7270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.207.65:46593 (error 108)
I20260929 18:33:06.018401 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 7481
I20260929 18:33:06.020172 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:06.019645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:46593 (local address 127.0.0.1:58946)
0929 18:33:06.019939 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:06.020032 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.65:46593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
W20260929 18:33:06.020569 7270 catalog_manager.cc:4584] TS 59fc421ff8aa44b28deedae788bb10e6 (127.28.207.65:46593): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 36a132f8994e469f8b9889c0c03554da: Network error: Client connection negotiation failed: client connection to 127.28.207.65:46593: connect: Connection refused (error 111)
I20260929 18:33:06.026047 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 7615
I20260929 18:33:06.033867 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 7253
2026-09-29T18:33:06Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2805 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:33:06 dist-test-slave-qcpv krb5kdc[7789](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:33:06 dist-test-slave-qcpv krb5kdc[7789](info): set up 2 sockets
Sep 29 18:33:06 dist-test-slave-qcpv krb5kdc[7789](info): commencing operation
krb5kdc: starting...
W20260929 18:33:07.144130 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:33:07 dist-test-slave-qcpv krb5kdc[7789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:33:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:33:07Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.207.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.207.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.207.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/krb5kdc/rangeradmin_127.28.207.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.207.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/krb5kdc/rangeradmin_127.28.207.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.207.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.207.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.207.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/krb5kdc/rangerlookup_127.28.207.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.207.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/krb5kdc/rangerlookup_127.28.207.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.207.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.207.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.207.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/krb5kdc/HTTP_127.28.207.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.207.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/krb5kdc/HTTP_127.28.207.84@KRBTEST.COM.keytab.
I20260929 18:33:07.360735 29501 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskxlLuiw/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/postgres -l logfile start
2026-09-29 18:33:10.507 UTC [7819] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 18:33:10.507 UTC [7819] LOG: listening on IPv4 address "127.28.207.84", port 50049
2026-09-29 18:33:10.512 UTC [7819] LOG: listening on Unix socket "/tmp/.s.PGSQL.50049"
2026-09-29 18:33:10.539 UTC [7822] LOG: database system was shut down at 2026-09-29 18:33:08 UTC
2026-09-29 18:33:10.544 UTC [7819] LOG: database system is ready to accept connections
I20260929 18:33:11.531834 29501 mini_postgres.cc:96] Postgres bound to 50049
2026-09-29 18:33:11.540 UTC [7830] FATAL: database "slave" does not exist
127.28.207.84:50049 - accepting connections
I20260929 18:33:11.541208 29501 mini_ranger.cc:161] Starting Ranger...
I20260929 18:33:11.559049 29501 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 18:33:12.212884 29501 mini_ranger.cc:87] Created ranger Postgres database
I20260929 18:33:12.213034 29501 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ranger-admin
2026-09-29 18:33:12,564 [I] DB FLAVOR :POSTGRES
2026-09-29 18:33:12,565 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 18:33:12,565 [I] Checking connection..
2026-09-29 18:33:12,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:50049/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 18:33:12,839 [I] Checking connection passed.
2026-09-29 18:33:12,839 [I] --------- Verifying version history table ---------
2026-09-29 18:33:12,839 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:50049/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 18:33:13,123 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 18:33:13,123 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:50049/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 18:33:13,394 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 18:33:13,395 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 18:33:13,395 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:50049/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 18:33:13.716 UTC [7894] WARNING: there is no transaction in progress
2026-09-29 18:33:13,724 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 18:33:13,725 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:50049/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 18:33:14,059 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 18:33:14,059 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 18:33:14,060 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:50049/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 18:33:14,332 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:50049/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 18:33:14,612 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:50049/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-qcpv.c.gcp-upstream.internal','N') ;"
2026-09-29 18:33:14,901 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:50049/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 18:33:15,191 [I] Table x_portal_user does not exist in database ranger
2026-09-29 18:33:15,192 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:50049/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 18:33:15,503 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 18:33:15,503 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:50049/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 18:33:15,770 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:50049/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 18:33:16,077 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 18:33:16,077 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:50049/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 18:33:17.271 UTC [8002] WARNING: there is no transaction in progress
2026-09-29 18:33:17.293 UTC [8002] WARNING: there is no transaction in progress
2026-09-29 18:33:17.315 UTC [8002] WARNING: there is no transaction in progress
2026-09-29 18:33:17.335 UTC [8002] WARNING: there is no transaction in progress
2026-09-29 18:33:17.355 UTC [8002] WARNING: there is no transaction in progress
2026-09-29 18:33:17.377 UTC [8002] WARNING: there is no transaction in progress
2026-09-29 18:33:17.442 UTC [8002] WARNING: there is no transaction in progress
2026-09-29 18:33:17.451 UTC [8002] WARNING: there is no transaction in progress
2026-09-29 18:33:17.468 UTC [8002] WARNING: there is no transaction in progress
2026-09-29 18:33:17.482 UTC [8002] WARNING: there is no transaction in progress
2026-09-29 18:33:17.494 UTC [8002] WARNING: there is no transaction in progress
2026-09-29 18:33:17.510 UTC [8002] WARNING: there is no transaction in progress
2026-09-29 18:33:17.523 UTC [8002] WARNING: there is no transaction in progress
2026-09-29 18:33:17.531 UTC [8002] WARNING: there is no transaction in progress
2026-09-29 18:33:17.540 UTC [8002] WARNING: there is no transaction in progress
2026-09-29 18:33:17.549 UTC [8002] WARNING: there is no transaction in progress
2026-09-29 18:33:18,424 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 18:33:18,424 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:50049/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 18:33:18,685 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 18:33:18,686 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:50049/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 18:33:18,987 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 18:33:18,987 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:50049/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 18:33:19,292 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 18:33:19,293 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:50049/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 18:33:19,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:50049/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 18:33:19,925 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:50049/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-qcpv.c.gcp-upstream.internal';"
2026-09-29 18:33:20,238 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 18:33:20,238 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:50049/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 18:33:20,525 [I] DB_PATCHES have already been applied
I20260929 18:33:20.532035 29501 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ranger-admin:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskxlLuiw/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 18:33:20.532150 29501 mini_ranger.cc:193] Using host: 127.28.207.84
I20260929 18:33:20.536718 29501 mini_ranger.cc:227] Ranger admin URL: http://127.28.207.84:39019
Sep 29, 2026 6:33:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 6:33:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 6:33:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 6:33:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 6:33:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 6:33:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 6:33:21 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 18:33:21.714527 29501 mini_ranger.cc:160] Time spent starting Ranger: real 10.173s user 0.002s sys 0.012s
Sep 29, 2026 6:33:46 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 6:33:46 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 6:33:46 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 6:33:46 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 6:33:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 6:33:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 6:33:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 6:33:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 6:33:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 6:33:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 6:33:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 6:33:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 6:33:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 6:33:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 6:33:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 6:33:46 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 6:33:46 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 6:33:47 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 18:33:48.118525 29501 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.207.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.207.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.207.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/krb5kdc/rangerkms_127.28.207.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.207.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/krb5kdc/rangerkms_127.28.207.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.207.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.207.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.207.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/krb5kdc/HTTP_127.28.207.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.207.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/krb5kdc/HTTP_127.28.207.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 18:33:48 dist-test-slave-qcpv krb5kdc[7789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706828, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 18:33:48.225960 29501 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 18:33:48.239292 29501 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 18:33:49.219149 29501 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 18:33:49.219267 29501 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ranger-kms
I20260929 18:33:49.219388 29501 mini_ranger_kms.cc:227] Using postgres at 127.28.207.84:50049
2026-09-29 18:33:49,419 [I] DB FLAVOR :POSTGRES
2026-09-29 18:33:49,419 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 18:33:49,420 [I] Checking connection
2026-09-29 18:33:49,420 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:50049/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 18:33:49,714 [I] connection success
2026-09-29 18:33:49,714 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 18:33:49,714 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 18:33:49,714 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:50049/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 18:33:50,015 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 18:33:50,015 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 18:33:50,015 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 18:33:50,015 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:50049/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 18:33:50,363 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 18:33:50.993490 29501 mini_ranger_kms.cc:310] Created kms service
I20260929 18:33:51.250540 29501 mini_ranger_kms.cc:326] Created kudu user
I20260929 18:33:51.309418 29501 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 18:33:51 dist-test-slave-qcpv krb5kdc[7789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706828, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.207.84@KRBTEST.COM
I20260929 18:33:51.860914 29501 mini_ranger_kms.cc:384] Added ranger policy
I20260929 18:33:51.861133 29501 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ranger-kms:/tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 18:33:51.861164 29501 mini_ranger_kms.cc:240] Using host: 127.28.207.84
I20260929 18:33:51.866567 29501 mini_ranger_kms.cc:276] Ranger KMS PID: 8183
I20260929 18:33:51.866811 29501 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.207.84:50695
Sep 29, 2026 6:33:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 6:33:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 6:33:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 6:33:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 6:33:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 6:33:53 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 18:33:54.427786 29501 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.202s user 0.002s sys 0.179s
I20260929 18:33:54.427981 29501 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 18:33:54.428149 29501 mini_ranger_kms.cc:401] 127.28.207.84:50695/kms/v1/keys
Sep 29 18:33:54 dist-test-slave-qcpv krb5kdc[7789](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790706834, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.207.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 18:33:54 dist-test-slave-qcpv krb5kdc[7789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706834, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.207.84@KRBTEST.COM for HTTP/127.28.207.84@KRBTEST.COM
Sep 29 18:33:55 dist-test-slave-qcpv krb5kdc[7789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706834, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.207.84@KRBTEST.COM for HTTP/127.28.207.84@KRBTEST.COM
Sep 29 18:33:58 dist-test-slave-qcpv krb5kdc[7789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.207.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
I20260929 18:33:58.263711 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:40307
--webserver_interface=127.28.207.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:44629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:40307
--ranger_config_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.207.84:50695/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:33:58.409224 8233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:33:58.409519 8233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:33:58.409561 8233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:33:58.413970 8233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:33:58.414052 8233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:33:58.414075 8233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:33:58.414093 8233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:33:58.414109 8233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:33:58.420218 8233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:44629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.207.84:50695/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:40307
--ranger_config_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:40307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:33:58.421429 8233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:33:58.422590 8233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:33:58.432343 8238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:58.432485 8233 server_base.cc:1034] running on GCE node
W20260929 18:33:58.432696 8239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:58.432772 8241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:58.433513 8233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:33:58.434788 8233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:33:58.436131 8233 hybrid_clock.cc:648] HybridClock initialized: now 1790706838436100 us; error 59 us; skew 500 ppm
Sep 29 18:33:58 dist-test-slave-qcpv krb5kdc[7789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706838, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:33:58.447075 8233 init.cc:377] Logged in from keytab as kudu/127.28.207.126@KRBTEST.COM (short username kudu)
I20260929 18:33:58.448271 8233 webserver.cc:460] Webserver started at http://127.28.207.126:43951/ using document root <none> and password file <none>
I20260929 18:33:58.448868 8233 fs_manager.cc:359] Metadata directory not provided
I20260929 18:33:58.448952 8233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:33:58.449182 8233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:33:58 dist-test-slave-qcpv krb5kdc[7789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706838, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for HTTP/127.28.207.84@KRBTEST.COM
I20260929 18:33:58.515686 8233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "080b9911896046dbb63c5f099ca192d0"
format_stamp: "Formatted at 2026-09-29 18:33:58 on dist-test-slave-qcpv"
server_key: "e369fa36b718c471e9aae04f422cc32f"
server_key_iv: "d95643c0ddda2ef3c08948129b131f05"
server_key_version: "kuduclusterkey@0"
I20260929 18:33:58.516410 8233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "080b9911896046dbb63c5f099ca192d0"
format_stamp: "Formatted at 2026-09-29 18:33:58 on dist-test-slave-qcpv"
server_key: "e369fa36b718c471e9aae04f422cc32f"
server_key_iv: "d95643c0ddda2ef3c08948129b131f05"
server_key_version: "kuduclusterkey@0"
I20260929 18:33:58.520797 8233 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:33:58.542066 8248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:58.543624 8233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260929 18:33:58.543780 8233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "080b9911896046dbb63c5f099ca192d0"
format_stamp: "Formatted at 2026-09-29 18:33:58 on dist-test-slave-qcpv"
server_key: "e369fa36b718c471e9aae04f422cc32f"
server_key_iv: "d95643c0ddda2ef3c08948129b131f05"
server_key_version: "kuduclusterkey@0"
I20260929 18:33:58.543931 8233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:33:58.556798 8233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:33:58.560288 8233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:33:58.560501 8233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:33:58.569394 8233 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:40307
I20260929 18:33:58.569489 8300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:40307 every 8 connection(s)
I20260929 18:33:58.570526 8233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:33:58.575016 8301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:33:58.575220 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 8233
I20260929 18:33:58.575364 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:33:58.575675 29501 external_mini_cluster.cc:1428] Setting key c943d01c9d32ee5bc380ca656806e905
I20260929 18:33:58.584678 8301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 080b9911896046dbb63c5f099ca192d0: Bootstrap starting.
Sep 29 18:33:58 dist-test-slave-qcpv krb5kdc[7789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:33:58.588120 8301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 080b9911896046dbb63c5f099ca192d0: Neither blocks nor log segments found. Creating new log.
I20260929 18:33:58.589897 8301 log.cc:824] T 00000000000000000000000000000000 P 080b9911896046dbb63c5f099ca192d0: Log is configured to *not* fsync() on all Append() calls
I20260929 18:33:58.590441 8304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:58.577318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52642 (local address 127.28.207.126:40307)
0929 18:33:58.577784 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:58.577809 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:58.577845 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:33:58.578570 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:58.578587 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:58.579041 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:58.579384 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:58.580157 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:58.581211 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:58.582075 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:58.582592 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:58.583013 (+ 421us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:58.585439 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:58.585445 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:58.585458 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:58.585489 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:58.587242 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:58.587681 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:58.587698 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:58.587703 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:58.587828 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:58.588119 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:58.588128 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:58.588130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:58.588626 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:58.588853 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:58.589137 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 18:33:58.589551 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":159,"threads_started":1}
I20260929 18:33:58.592868 8301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 080b9911896046dbb63c5f099ca192d0: No bootstrap required, opened a new log
I20260929 18:33:58.597676 8301 raft_consensus.cc:348] T 00000000000000000000000000000000 P 080b9911896046dbb63c5f099ca192d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080b9911896046dbb63c5f099ca192d0" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 40307 } }
I20260929 18:33:58.598018 8301 raft_consensus.cc:374] T 00000000000000000000000000000000 P 080b9911896046dbb63c5f099ca192d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:33:58.598086 8301 raft_consensus.cc:729] T 00000000000000000000000000000000 P 080b9911896046dbb63c5f099ca192d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 080b9911896046dbb63c5f099ca192d0, State: Initialized, Role: FOLLOWER
I20260929 18:33:58.598642 8301 consensus_queue.cc:260] T 00000000000000000000000000000000 P 080b9911896046dbb63c5f099ca192d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080b9911896046dbb63c5f099ca192d0" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 40307 } }
I20260929 18:33:58.598831 8301 raft_consensus.cc:388] T 00000000000000000000000000000000 P 080b9911896046dbb63c5f099ca192d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:33:58.598944 8301 raft_consensus.cc:482] T 00000000000000000000000000000000 P 080b9911896046dbb63c5f099ca192d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:33:58.599053 8301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 080b9911896046dbb63c5f099ca192d0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:33:58.600449 8301 raft_consensus.cc:504] T 00000000000000000000000000000000 P 080b9911896046dbb63c5f099ca192d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080b9911896046dbb63c5f099ca192d0" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 40307 } }
I20260929 18:33:58.600873 8301 leader_election.cc:302] T 00000000000000000000000000000000 P 080b9911896046dbb63c5f099ca192d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 080b9911896046dbb63c5f099ca192d0; no voters:
I20260929 18:33:58.601302 8301 leader_election.cc:288] T 00000000000000000000000000000000 P 080b9911896046dbb63c5f099ca192d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:33:58.601476 8306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 080b9911896046dbb63c5f099ca192d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:33:58.601926 8306 raft_consensus.cc:686] T 00000000000000000000000000000000 P 080b9911896046dbb63c5f099ca192d0 [term 1 LEADER]: Becoming Leader. State: Replica: 080b9911896046dbb63c5f099ca192d0, State: Running, Role: LEADER
I20260929 18:33:58.602418 8306 consensus_queue.cc:237] T 00000000000000000000000000000000 P 080b9911896046dbb63c5f099ca192d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080b9911896046dbb63c5f099ca192d0" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 40307 } }
I20260929 18:33:58.603156 8301 sys_catalog.cc:564] T 00000000000000000000000000000000 P 080b9911896046dbb63c5f099ca192d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:33:58.603886 8307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 080b9911896046dbb63c5f099ca192d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 080b9911896046dbb63c5f099ca192d0. Latest consensus state: current_term: 1 leader_uuid: "080b9911896046dbb63c5f099ca192d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080b9911896046dbb63c5f099ca192d0" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 40307 } } }
I20260929 18:33:58.604018 8307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 080b9911896046dbb63c5f099ca192d0 [sys.catalog]: This master's current role is: LEADER
I20260929 18:33:58.605896 8308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 080b9911896046dbb63c5f099ca192d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "080b9911896046dbb63c5f099ca192d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "080b9911896046dbb63c5f099ca192d0" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 40307 } } }
I20260929 18:33:58.606032 8308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 080b9911896046dbb63c5f099ca192d0 [sys.catalog]: This master's current role is: LEADER
I20260929 18:33:58.605890 8312 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:33:58.610164 8312 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:33:58.610671 8301 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 18:33:58.623346 8312 catalog_manager.cc:1347] Generated new cluster ID: 6c5ba67249204a3095006bdea1bdd4ed
I20260929 18:33:58.623445 8312 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:33:58.634253 8312 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:33:58.636088 8312 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:33:58.646651 8312 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 080b9911896046dbb63c5f099ca192d0: Generated new TSK 0
I20260929 18:33:58.647840 8312 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.207.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab.
Entry for principal kudu/127.28.207.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab.
I20260929 18:33:59.395840 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.207.84:50695/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.65:0
--local_ip_for_outbound_sockets=127.28.207.65
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:40307
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:44629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:33:59.559083 8346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:33:59.559551 8346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:33:59.559624 8346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:33:59.566046 8346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:33:59.566146 8346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:33:59.566263 8346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.65
I20260929 18:33:59.575093 8346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:44629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.207.84:50695/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:40307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.65
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:33:59.577186 8346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:33:59.580724 8346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:33:59.597996 8356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:59.598192 8353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:59.598019 8354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:59.599831 8346 server_base.cc:1034] running on GCE node
I20260929 18:33:59.600520 8346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:33:59.601579 8346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:33:59.602805 8346 hybrid_clock.cc:648] HybridClock initialized: now 1790706839602776 us; error 48 us; skew 500 ppm
Sep 29 18:33:59 dist-test-slave-qcpv krb5kdc[7789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706839, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:33:59.608274 8346 init.cc:377] Logged in from keytab as kudu/127.28.207.65@KRBTEST.COM (short username kudu)
I20260929 18:33:59.609855 8346 webserver.cc:460] Webserver started at http://127.28.207.65:39897/ using document root <none> and password file <none>
I20260929 18:33:59.610800 8346 fs_manager.cc:359] Metadata directory not provided
I20260929 18:33:59.610949 8346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:33:59.611234 8346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:33:59 dist-test-slave-qcpv krb5kdc[7789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706839, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for HTTP/127.28.207.84@KRBTEST.COM
I20260929 18:33:59.639096 8346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-0/data/instance:
uuid: "e5bc0d9b3c7a47bd949d9f98d0ca2be6"
format_stamp: "Formatted at 2026-09-29 18:33:59 on dist-test-slave-qcpv"
server_key: "a3cd647333ad6bb44e9c7e87dae34cf2"
server_key_iv: "2cd1792478e5c8b8833d850c935849ca"
server_key_version: "kuduclusterkey@0"
I20260929 18:33:59.639837 8346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance:
uuid: "e5bc0d9b3c7a47bd949d9f98d0ca2be6"
format_stamp: "Formatted at 2026-09-29 18:33:59 on dist-test-slave-qcpv"
server_key: "a3cd647333ad6bb44e9c7e87dae34cf2"
server_key_iv: "2cd1792478e5c8b8833d850c935849ca"
server_key_version: "kuduclusterkey@0"
I20260929 18:33:59.645114 8346 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 18:33:59.676378 8365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:59.678026 8346 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:33:59.678187 8346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-0/wal
uuid: "e5bc0d9b3c7a47bd949d9f98d0ca2be6"
format_stamp: "Formatted at 2026-09-29 18:33:59 on dist-test-slave-qcpv"
server_key: "a3cd647333ad6bb44e9c7e87dae34cf2"
server_key_iv: "2cd1792478e5c8b8833d850c935849ca"
server_key_version: "kuduclusterkey@0"
I20260929 18:33:59.678457 8346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:33:59.703136 8346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:33:59.707512 8346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:33:59.707736 8346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:33:59.708566 8346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:33:59.709745 8346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:33:59.709820 8346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:33:59.709877 8346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:33:59.709985 8346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260929 18:33:59.725206 8346 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.65:45317
I20260929 18:33:59.726574 8346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
I20260929 18:33:59.727275 8478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.65:45317 every 8 connection(s)
I20260929 18:33:59.729590 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 8346
I20260929 18:33:59.729717 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance
I20260929 18:33:59.730057 29501 external_mini_cluster.cc:1428] Setting key 89e74e591987419e64b654adf0c966d8
Sep 29 18:33:59 dist-test-slave-qcpv krb5kdc[7789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706839, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.207.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.66@KRBTEST.COM" created.
I20260929 18:33:59.749408 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:59.732151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:45549 (local address 127.28.207.126:40307)
0929 18:33:59.732496 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0929 18:33:59.732501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:33:59.733025 (+ 524us) server_negotiation.cc:407] Connection header received
0929 18:33:59.734483 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:33:59.734487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:33:59.734554 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:33:59.734628 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:33:59.736614 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:59.737578 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:59.739519 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:33:59.739916 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:33:59.739999 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:59.743660 (+ 3661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:33:59.743665 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:33:59.743668 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:33:59.743700 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:33:59.746026 (+ 2326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:59.747005 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:59.747009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:59.747011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:59.747114 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:33:59.747657 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:33:59.747661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:33:59.747663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:33:59.747936 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:33:59.748044 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 18:33:59.749000 (+ 956us) server_negotiation.cc:299] Negotiation successful
0929 18:33:59.749166 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":123,"threads_started":1}
I20260929 18:33:59.751461 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:33:59.730821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:40307 (local address 127.28.207.65:45549)
0929 18:33:59.732826 (+ 2005us) negotiation.cc:107] Waiting for socket to connect
0929 18:33:59.732896 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0929 18:33:59.734196 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:33:59.735184 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:33:59.735194 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:33:59.735630 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:33:59.736424 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:59.736445 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:59.737749 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:59.737753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:59.739346 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:33:59.739357 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:33:59.740253 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:33:59.740257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:33:59.740532 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:33:59.741421 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:33:59.741444 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:33:59.743427 (+ 1983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:33:59.746263 (+ 2836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:59.746271 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:59.746345 (+ 74us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:59.746765 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:59.747265 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:33:59.747270 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:33:59.747272 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:33:59.747544 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:33:59.748138 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:33:59.748146 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:33:59.748679 (+ 533us) client_negotiation.cc:769] Sending connection context
0929 18:33:59.749693 (+ 1014us) client_negotiation.cc:240] Negotiation successful
0929 18:33:59.750129 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1699,"thread_start_us":181,"threads_started":1}
I20260929 18:33:59.753363 8479 heartbeater.cc:347] Connected to a master server at 127.28.207.126:40307
I20260929 18:33:59.753664 8479 heartbeater.cc:464] Registering TS with master...
I20260929 18:33:59.754347 8479 heartbeater.cc:511] Master 127.28.207.126:40307 requested a full tablet report, sending...
I20260929 18:33:59.756349 8259 ts_manager.cc:194] Registered new tserver with Master: e5bc0d9b3c7a47bd949d9f98d0ca2be6 (127.28.207.65:45317)
I20260929 18:33:59.757925 8259 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.65@KRBTEST.COM'} at 127.28.207.65:45549
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab.
Entry for principal kudu/127.28.207.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab.
I20260929 18:33:59.775512 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.207.84:50695/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.66:0
--local_ip_for_outbound_sockets=127.28.207.66
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:40307
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:44629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:33:59 dist-test-slave-qcpv krb5kdc[7789](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790706839, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 18:33:59.928997 8485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:33:59.929411 8485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:33:59.929518 8485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:33:59.933737 8485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:33:59.933827 8485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:33:59.933929 8485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.66
I20260929 18:33:59.939608 8485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:44629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.207.84:50695/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:40307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.66
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:33:59.940884 8485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:33:59.942137 8485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:33:59.954097 8494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:59.954089 8491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:33:59.954385 8492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:33:59.955739 8485 server_base.cc:1034] running on GCE node
I20260929 18:33:59.956256 8485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:33:59.957010 8485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:33:59.958236 8485 hybrid_clock.cc:648] HybridClock initialized: now 1790706839958172 us; error 49 us; skew 500 ppm
Sep 29 18:33:59 dist-test-slave-qcpv krb5kdc[7789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706839, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:33:59.962033 8485 init.cc:377] Logged in from keytab as kudu/127.28.207.66@KRBTEST.COM (short username kudu)
I20260929 18:33:59.963281 8485 webserver.cc:460] Webserver started at http://127.28.207.66:36451/ using document root <none> and password file <none>
I20260929 18:33:59.963999 8485 fs_manager.cc:359] Metadata directory not provided
I20260929 18:33:59.964072 8485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:33:59.964294 8485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:33:59 dist-test-slave-qcpv krb5kdc[7789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706839, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for HTTP/127.28.207.84@KRBTEST.COM
I20260929 18:33:59.980993 8485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-1/data/instance:
uuid: "39374b51e8164b0b91d32a2d68bb97df"
format_stamp: "Formatted at 2026-09-29 18:33:59 on dist-test-slave-qcpv"
server_key: "7dbd891a48c7774cef551ff03b2652b7"
server_key_iv: "e4541c34a34fa44f4258b457b56f0838"
server_key_version: "kuduclusterkey@0"
I20260929 18:33:59.982151 8485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance:
uuid: "39374b51e8164b0b91d32a2d68bb97df"
format_stamp: "Formatted at 2026-09-29 18:33:59 on dist-test-slave-qcpv"
server_key: "7dbd891a48c7774cef551ff03b2652b7"
server_key_iv: "e4541c34a34fa44f4258b457b56f0838"
server_key_version: "kuduclusterkey@0"
I20260929 18:33:59.987586 8485 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 18:34:00.004855 8501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:00.006733 8485 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:34:00.007052 8485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-1/wal
uuid: "39374b51e8164b0b91d32a2d68bb97df"
format_stamp: "Formatted at 2026-09-29 18:33:59 on dist-test-slave-qcpv"
server_key: "7dbd891a48c7774cef551ff03b2652b7"
server_key_iv: "e4541c34a34fa44f4258b457b56f0838"
server_key_version: "kuduclusterkey@0"
I20260929 18:34:00.007225 8485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:34:00.022974 8485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:34:00.026568 8485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:34:00.026800 8485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:34:00.027489 8485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:34:00.028458 8485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:34:00.028528 8485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:00.028594 8485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:34:00.028649 8485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:00.041664 8485 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.66:46639
I20260929 18:34:00.047014 8614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.66:46639 every 8 connection(s)
I20260929 18:34:00.047972 8485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
I20260929 18:34:00.056605 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 8485
I20260929 18:34:00.056764 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance
I20260929 18:34:00.057106 29501 external_mini_cluster.cc:1428] Setting key 5797a33062ed5d66c57f35da110c789d
Sep 29 18:34:00 dist-test-slave-qcpv krb5kdc[7789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706839, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.207.67@KRBTEST.COM; defaulting to no policy
I20260929 18:34:00.074402 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:00.051254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:48385 (local address 127.28.207.126:40307)
0929 18:34:00.051420 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:00.051425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:00.055665 (+ 4240us) server_negotiation.cc:407] Connection header received
0929 18:34:00.057402 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:00.057408 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:00.057488 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:00.057560 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:34:00.060268 (+ 2708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:00.061188 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:00.065038 (+ 3850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:00.065359 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:00.065434 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:00.069363 (+ 3929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:34:00.069368 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:34:00.069371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:34:00.069405 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:34:00.071524 (+ 2119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:00.072309 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:00.072313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:00.072315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:00.072403 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:00.072849 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:00.072852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:00.072854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:00.073115 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:34:00.073225 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:00.074023 (+ 798us) server_negotiation.cc:299] Negotiation successful
0929 18:34:00.074200 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.67@KRBTEST.COM" created.
I20260929 18:34:00.077019 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:00.051708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:40307 (local address 127.28.207.66:48385)
0929 18:34:00.055328 (+ 3620us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:00.055445 (+ 117us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:00.057021 (+ 1576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:00.058462 (+ 1441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:00.058477 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:00.059196 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:34:00.060078 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:00.060100 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:00.061366 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:00.061369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:00.062949 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:00.062964 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:00.065676 (+ 2712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:00.065679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:00.065873 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:00.066780 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:34:00.066810 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:34:00.069078 (+ 2268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:34:00.071737 (+ 2659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:34:00.071749 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:34:00.071774 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:34:00.072182 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:34:00.072509 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:34:00.072515 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:34:00.072519 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:34:00.072724 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:34:00.073301 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:34:00.073309 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:34:00.073712 (+ 403us) client_negotiation.cc:769] Sending connection context
0929 18:34:00.075206 (+ 1494us) client_negotiation.cc:240] Negotiation successful
0929 18:34:00.075647 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3230,"thread_start_us":136,"threads_started":1}
I20260929 18:34:00.078734 8615 heartbeater.cc:347] Connected to a master server at 127.28.207.126:40307
I20260929 18:34:00.079131 8615 heartbeater.cc:464] Registering TS with master...
I20260929 18:34:00.079895 8615 heartbeater.cc:511] Master 127.28.207.126:40307 requested a full tablet report, sending...
I20260929 18:34:00.082151 8259 ts_manager.cc:194] Registered new tserver with Master: 39374b51e8164b0b91d32a2d68bb97df (127.28.207.66:46639)
I20260929 18:34:00.083062 8259 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.66@KRBTEST.COM'} at 127.28.207.66:48385
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab.
Entry for principal kudu/127.28.207.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab.
I20260929 18:34:00.099118 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.207.84:50695/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.67:0
--local_ip_for_outbound_sockets=127.28.207.67
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:40307
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:44629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:34:00.287618 8620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:34:00.288206 8620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:34:00.288367 8620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:34:00.294936 8620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:34:00.295128 8620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:34:00.295307 8620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.67
I20260929 18:34:00.304543 8620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:44629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.207.84:50695/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:40307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.67
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:34:00.306743 8620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:34:00.308387 8620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 29 18:34:00 dist-test-slave-qcpv krb5kdc[7789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706839, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for HTTP/127.28.207.84@KRBTEST.COM
W20260929 18:34:00.324268 8626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:00.324683 8625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:00.324800 8628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:34:00.326972 8620 server_base.cc:1034] running on GCE node
I20260929 18:34:00.327571 8620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:34:00.328539 8620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:34:00.344488 8620 hybrid_clock.cc:648] HybridClock initialized: now 1790706840344321 us; error 90 us; skew 500 ppm
Sep 29 18:34:00 dist-test-slave-qcpv krb5kdc[7789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706840, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:34:00.351120 8620 init.cc:377] Logged in from keytab as kudu/127.28.207.67@KRBTEST.COM (short username kudu)
I20260929 18:34:00.352811 8620 webserver.cc:460] Webserver started at http://127.28.207.67:46679/ using document root <none> and password file <none>
I20260929 18:34:00.354085 8620 fs_manager.cc:359] Metadata directory not provided
I20260929 18:34:00.354243 8620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:34:00.354521 8620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:34:00 dist-test-slave-qcpv krb5kdc[7789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706840, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for HTTP/127.28.207.84@KRBTEST.COM
I20260929 18:34:00.393235 8620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-2/data/instance:
uuid: "09fb1f3a5b7c4d64924559227b5b0e3b"
format_stamp: "Formatted at 2026-09-29 18:34:00 on dist-test-slave-qcpv"
server_key: "979936890e5bae7688b49659254195d2"
server_key_iv: "146238ae379d874463f248f924aa502e"
server_key_version: "kuduclusterkey@0"
I20260929 18:34:00.396238 8620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance:
uuid: "09fb1f3a5b7c4d64924559227b5b0e3b"
format_stamp: "Formatted at 2026-09-29 18:34:00 on dist-test-slave-qcpv"
server_key: "979936890e5bae7688b49659254195d2"
server_key_iv: "146238ae379d874463f248f924aa502e"
server_key_version: "kuduclusterkey@0"
I20260929 18:34:00.409157 8620 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.004s sys 0.000s
I20260929 18:34:00.432987 8635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:00.435032 8620 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 18:34:00.435235 8620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-2/wal
uuid: "09fb1f3a5b7c4d64924559227b5b0e3b"
format_stamp: "Formatted at 2026-09-29 18:34:00 on dist-test-slave-qcpv"
server_key: "979936890e5bae7688b49659254195d2"
server_key_iv: "146238ae379d874463f248f924aa502e"
server_key_version: "kuduclusterkey@0"
I20260929 18:34:00.435391 8620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:34:00.474879 8620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:34:00.479013 8620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:34:00.479250 8620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:34:00.480049 8620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:34:00.481236 8620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:34:00.481309 8620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:00.481370 8620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:34:00.481415 8620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:00.494540 8620 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.67:37273
I20260929 18:34:00.495738 8620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
I20260929 18:34:00.497058 8748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.67:37273 every 8 connection(s)
I20260929 18:34:00.505471 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 8620
I20260929 18:34:00.505631 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance
I20260929 18:34:00.505976 29501 external_mini_cluster.cc:1428] Setting key bdb31ca32471845ca29ebc730f6bbff8
Sep 29 18:34:00 dist-test-slave-qcpv krb5kdc[7789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706840, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:34:00.514705 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:00.499145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:35591 (local address 127.28.207.126:40307)
0929 18:34:00.499336 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:00.499342 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:00.499906 (+ 564us) server_negotiation.cc:407] Connection header received
0929 18:34:00.501449 (+ 1543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:00.501455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:00.501531 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:00.501659 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:34:00.503459 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:00.504322 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:00.506311 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:00.506689 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:00.506769 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:00.510032 (+ 3263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:34:00.510037 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:34:00.510040 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:34:00.510069 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:34:00.511854 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:00.512718 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:00.512721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:00.512723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:00.512816 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:00.513296 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:00.513299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:00.513301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:00.513527 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:34:00.513629 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:00.514395 (+ 766us) server_negotiation.cc:299] Negotiation successful
0929 18:34:00.514534 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 18:34:00.516362 8751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:00.498994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:40307 (local address 127.28.207.67:35591)
0929 18:34:00.499692 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:00.499770 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:00.501116 (+ 1346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:00.501927 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:00.501938 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:00.502394 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:34:00.503268 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:00.503291 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:00.504586 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:00.504593 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:00.506157 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:00.506172 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:00.507074 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:00.507078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:00.507327 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:00.508081 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:34:00.508103 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:34:00.509808 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:34:00.512057 (+ 2249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:34:00.512064 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:34:00.512084 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:34:00.512535 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:34:00.512933 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:34:00.512936 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:34:00.512938 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:34:00.513152 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:34:00.513660 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:34:00.513668 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:34:00.514056 (+ 388us) client_negotiation.cc:769] Sending connection context
0929 18:34:00.514297 (+ 241us) client_negotiation.cc:240] Negotiation successful
0929 18:34:00.514712 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":163,"threads_started":1}
I20260929 18:34:00.518567 8749 heartbeater.cc:347] Connected to a master server at 127.28.207.126:40307
I20260929 18:34:00.518944 8749 heartbeater.cc:464] Registering TS with master...
I20260929 18:34:00.519589 8749 heartbeater.cc:511] Master 127.28.207.126:40307 requested a full tablet report, sending...
I20260929 18:34:00.520990 8259 ts_manager.cc:194] Registered new tserver with Master: 09fb1f3a5b7c4d64924559227b5b0e3b (127.28.207.67:37273)
I20260929 18:34:00.521867 8259 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.67@KRBTEST.COM'} at 127.28.207.67:35591
I20260929 18:34:00.532821 29501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:34:00.549400 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:00.535769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52654 (local address 127.28.207.126:40307)
0929 18:34:00.535956 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:00.535961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:00.539652 (+ 3691us) server_negotiation.cc:407] Connection header received
0929 18:34:00.539821 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:00.539824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:00.539895 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:00.540004 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:34:00.541049 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:00.541963 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:00.543300 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:00.543657 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:00.543767 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:00.545762 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:34:00.545766 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:34:00.545769 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:34:00.545796 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:34:00.547504 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:00.548029 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:00.548033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:00.548035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:00.548120 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:00.548422 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:00.548426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:00.548428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:00.548644 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:34:00.548743 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:00.549103 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 18:34:00.549230 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 18:34:00.554354 8259 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52654:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:34:00.559155 8259 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:34:00.583370 8762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:00.571981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:45317 (local address 127.0.0.1:48868)
0929 18:34:00.573945 (+ 1964us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:00.574011 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:00.574184 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:00.576388 (+ 2204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:00.576391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:00.576412 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:34:00.576533 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:00.576541 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:00.578047 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:00.578051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:00.580253 (+ 2202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:00.580264 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:00.581767 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:00.581771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:00.582828 (+ 1057us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:00.582932 (+ 104us) client_negotiation.cc:769] Sending connection context
0929 18:34:00.583061 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 18:34:00.583203 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1743,"mutex_wait_us":82,"thread_start_us":142,"threads_started":1}
I20260929 18:34:00.584348 8764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:00.572224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:37273 (local address 127.0.0.1:38916)
0929 18:34:00.574305 (+ 2081us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:00.574322 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:00.574420 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:00.575490 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:00.575496 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:00.575519 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:34:00.575659 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:00.575666 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:00.577103 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:00.577107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:00.578513 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:00.578522 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:00.582531 (+ 4009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:00.582535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:00.582709 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:00.584089 (+ 1380us) client_negotiation.cc:769] Sending connection context
0929 18:34:00.584154 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 18:34:00.584229 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2008,"mutex_wait_us":1342,"thread_start_us":103,"threads_started":1}
I20260929 18:34:00.584950 8765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:00.573151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48868 (local address 127.28.207.65:45317)
0929 18:34:00.574501 (+ 1350us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:00.574509 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:00.574535 (+ 26us) server_negotiation.cc:407] Connection header received
0929 18:34:00.574626 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:00.574631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:00.574932 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:00.575103 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:34:00.576683 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:00.577898 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:00.580410 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:00.581376 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:00.581501 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:00.581696 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:00.584637 (+ 2941us) server_negotiation.cc:299] Negotiation successful
0929 18:34:00.584794 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1233,"thread_start_us":131,"threads_started":1}
I20260929 18:34:00.585678 8766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:00.573499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38916 (local address 127.28.207.67:37273)
0929 18:34:00.574648 (+ 1149us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:00.574659 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:00.574681 (+ 22us) server_negotiation.cc:407] Connection header received
0929 18:34:00.574758 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:00.574763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:00.575008 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:00.575153 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:34:00.575778 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:00.576977 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:00.581364 (+ 4387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:00.582209 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:00.582313 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:00.582465 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:00.585433 (+ 2968us) server_negotiation.cc:299] Negotiation successful
0929 18:34:00.585539 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1054,"thread_start_us":139,"threads_started":1}
I20260929 18:34:00.589709 8413 tablet_service.cc:1467] Processing CreateTablet for tablet 41653766a021454f80a515eff3e7979b (DEFAULT_TABLE table=test-table [id=9a9a6be5d7be4a4f9892c8159cd7b6ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:34:00.591248 8413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41653766a021454f80a515eff3e7979b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:34:00.593290 8683 tablet_service.cc:1467] Processing CreateTablet for tablet 41653766a021454f80a515eff3e7979b (DEFAULT_TABLE table=test-table [id=9a9a6be5d7be4a4f9892c8159cd7b6ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:34:00.594378 8683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41653766a021454f80a515eff3e7979b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:34:00.595582 8763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:00.572975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:46639 (local address 127.0.0.1:39308)
0929 18:34:00.578743 (+ 5768us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:00.578756 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:00.578831 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:00.579549 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:00.579552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:00.579570 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:34:00.579671 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:00.579677 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:00.581044 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:00.581047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:00.583733 (+ 2686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:00.583745 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:00.595026 (+ 11281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:00.595032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:00.595170 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:00.595204 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 18:34:00.595307 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 18:34:00.595405 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5704,"mutex_wait_us":2,"thread_start_us":91,"threads_started":1}
I20260929 18:34:00.596957 8767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:00.573858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39308 (local address 127.28.207.66:46639)
0929 18:34:00.575409 (+ 1551us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:00.575422 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:00.578925 (+ 3503us) server_negotiation.cc:407] Connection header received
0929 18:34:00.579064 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:00.579069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:00.579264 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:00.579410 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:34:00.579796 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:00.580913 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:00.583908 (+ 2995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:00.585083 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:00.585285 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:00.596591 (+ 11306us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:00.596663 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 18:34:00.596791 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1454,"thread_start_us":83,"threads_started":1}
I20260929 18:34:00.601728 8533 tablet_service.cc:1467] Processing CreateTablet for tablet 41653766a021454f80a515eff3e7979b (DEFAULT_TABLE table=test-table [id=9a9a6be5d7be4a4f9892c8159cd7b6ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:34:00.602994 8533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41653766a021454f80a515eff3e7979b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:34:00.607023 8768 tablet_bootstrap.cc:492] T 41653766a021454f80a515eff3e7979b P 09fb1f3a5b7c4d64924559227b5b0e3b: Bootstrap starting.
I20260929 18:34:00.610169 8768 tablet_bootstrap.cc:654] T 41653766a021454f80a515eff3e7979b P 09fb1f3a5b7c4d64924559227b5b0e3b: Neither blocks nor log segments found. Creating new log.
I20260929 18:34:00.611156 8768 log.cc:824] T 41653766a021454f80a515eff3e7979b P 09fb1f3a5b7c4d64924559227b5b0e3b: Log is configured to *not* fsync() on all Append() calls
I20260929 18:34:00.614082 8769 tablet_bootstrap.cc:492] T 41653766a021454f80a515eff3e7979b P 39374b51e8164b0b91d32a2d68bb97df: Bootstrap starting.
I20260929 18:34:00.614641 8768 tablet_bootstrap.cc:492] T 41653766a021454f80a515eff3e7979b P 09fb1f3a5b7c4d64924559227b5b0e3b: No bootstrap required, opened a new log
I20260929 18:34:00.614892 8768 ts_tablet_manager.cc:1397] T 41653766a021454f80a515eff3e7979b P 09fb1f3a5b7c4d64924559227b5b0e3b: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 18:34:00.616842 8769 tablet_bootstrap.cc:654] T 41653766a021454f80a515eff3e7979b P 39374b51e8164b0b91d32a2d68bb97df: Neither blocks nor log segments found. Creating new log.
I20260929 18:34:00.617785 8769 log.cc:824] T 41653766a021454f80a515eff3e7979b P 39374b51e8164b0b91d32a2d68bb97df: Log is configured to *not* fsync() on all Append() calls
I20260929 18:34:00.618765 8771 tablet_bootstrap.cc:492] T 41653766a021454f80a515eff3e7979b P e5bc0d9b3c7a47bd949d9f98d0ca2be6: Bootstrap starting.
I20260929 18:34:00.619589 8768 raft_consensus.cc:348] T 41653766a021454f80a515eff3e7979b P 09fb1f3a5b7c4d64924559227b5b0e3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5bc0d9b3c7a47bd949d9f98d0ca2be6" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 45317 } } peers { permanent_uuid: "09fb1f3a5b7c4d64924559227b5b0e3b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37273 } } peers { permanent_uuid: "39374b51e8164b0b91d32a2d68bb97df" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46639 } }
I20260929 18:34:00.619956 8768 raft_consensus.cc:374] T 41653766a021454f80a515eff3e7979b P 09fb1f3a5b7c4d64924559227b5b0e3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:34:00.620007 8768 raft_consensus.cc:729] T 41653766a021454f80a515eff3e7979b P 09fb1f3a5b7c4d64924559227b5b0e3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09fb1f3a5b7c4d64924559227b5b0e3b, State: Initialized, Role: FOLLOWER
I20260929 18:34:00.621310 8771 tablet_bootstrap.cc:654] T 41653766a021454f80a515eff3e7979b P e5bc0d9b3c7a47bd949d9f98d0ca2be6: Neither blocks nor log segments found. Creating new log.
I20260929 18:34:00.621748 8768 consensus_queue.cc:260] T 41653766a021454f80a515eff3e7979b P 09fb1f3a5b7c4d64924559227b5b0e3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5bc0d9b3c7a47bd949d9f98d0ca2be6" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 45317 } } peers { permanent_uuid: "09fb1f3a5b7c4d64924559227b5b0e3b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37273 } } peers { permanent_uuid: "39374b51e8164b0b91d32a2d68bb97df" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46639 } }
I20260929 18:34:00.622241 8771 log.cc:824] T 41653766a021454f80a515eff3e7979b P e5bc0d9b3c7a47bd949d9f98d0ca2be6: Log is configured to *not* fsync() on all Append() calls
I20260929 18:34:00.623039 8768 ts_tablet_manager.cc:1428] T 41653766a021454f80a515eff3e7979b P 09fb1f3a5b7c4d64924559227b5b0e3b: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260929 18:34:00.623553 8749 heartbeater.cc:503] Master 127.28.207.126:40307 was elected leader, sending a full tablet report...
I20260929 18:34:00.624946 8769 tablet_bootstrap.cc:492] T 41653766a021454f80a515eff3e7979b P 39374b51e8164b0b91d32a2d68bb97df: No bootstrap required, opened a new log
I20260929 18:34:00.624944 8771 tablet_bootstrap.cc:492] T 41653766a021454f80a515eff3e7979b P e5bc0d9b3c7a47bd949d9f98d0ca2be6: No bootstrap required, opened a new log
I20260929 18:34:00.625149 8771 ts_tablet_manager.cc:1397] T 41653766a021454f80a515eff3e7979b P e5bc0d9b3c7a47bd949d9f98d0ca2be6: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260929 18:34:00.625149 8769 ts_tablet_manager.cc:1397] T 41653766a021454f80a515eff3e7979b P 39374b51e8164b0b91d32a2d68bb97df: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260929 18:34:00.631031 8769 raft_consensus.cc:348] T 41653766a021454f80a515eff3e7979b P 39374b51e8164b0b91d32a2d68bb97df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5bc0d9b3c7a47bd949d9f98d0ca2be6" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 45317 } } peers { permanent_uuid: "09fb1f3a5b7c4d64924559227b5b0e3b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37273 } } peers { permanent_uuid: "39374b51e8164b0b91d32a2d68bb97df" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46639 } }
I20260929 18:34:00.631014 8771 raft_consensus.cc:348] T 41653766a021454f80a515eff3e7979b P e5bc0d9b3c7a47bd949d9f98d0ca2be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5bc0d9b3c7a47bd949d9f98d0ca2be6" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 45317 } } peers { permanent_uuid: "09fb1f3a5b7c4d64924559227b5b0e3b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37273 } } peers { permanent_uuid: "39374b51e8164b0b91d32a2d68bb97df" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46639 } }
I20260929 18:34:00.631340 8769 raft_consensus.cc:374] T 41653766a021454f80a515eff3e7979b P 39374b51e8164b0b91d32a2d68bb97df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:34:00.631408 8769 raft_consensus.cc:729] T 41653766a021454f80a515eff3e7979b P 39374b51e8164b0b91d32a2d68bb97df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39374b51e8164b0b91d32a2d68bb97df, State: Initialized, Role: FOLLOWER
I20260929 18:34:00.631961 8769 consensus_queue.cc:260] T 41653766a021454f80a515eff3e7979b P 39374b51e8164b0b91d32a2d68bb97df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5bc0d9b3c7a47bd949d9f98d0ca2be6" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 45317 } } peers { permanent_uuid: "09fb1f3a5b7c4d64924559227b5b0e3b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37273 } } peers { permanent_uuid: "39374b51e8164b0b91d32a2d68bb97df" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46639 } }
I20260929 18:34:00.631340 8771 raft_consensus.cc:374] T 41653766a021454f80a515eff3e7979b P e5bc0d9b3c7a47bd949d9f98d0ca2be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:34:00.633299 8769 ts_tablet_manager.cc:1428] T 41653766a021454f80a515eff3e7979b P 39374b51e8164b0b91d32a2d68bb97df: Time spent starting tablet: real 0.008s user 0.005s sys 0.005s
I20260929 18:34:00.633719 8615 heartbeater.cc:503] Master 127.28.207.126:40307 was elected leader, sending a full tablet report...
I20260929 18:34:00.632723 8771 raft_consensus.cc:729] T 41653766a021454f80a515eff3e7979b P e5bc0d9b3c7a47bd949d9f98d0ca2be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5bc0d9b3c7a47bd949d9f98d0ca2be6, State: Initialized, Role: FOLLOWER
I20260929 18:34:00.634507 8771 consensus_queue.cc:260] T 41653766a021454f80a515eff3e7979b P e5bc0d9b3c7a47bd949d9f98d0ca2be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5bc0d9b3c7a47bd949d9f98d0ca2be6" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 45317 } } peers { permanent_uuid: "09fb1f3a5b7c4d64924559227b5b0e3b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37273 } } peers { permanent_uuid: "39374b51e8164b0b91d32a2d68bb97df" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46639 } }
I20260929 18:34:00.635740 8771 ts_tablet_manager.cc:1428] T 41653766a021454f80a515eff3e7979b P e5bc0d9b3c7a47bd949d9f98d0ca2be6: Time spent starting tablet: real 0.011s user 0.004s sys 0.005s
I20260929 18:34:00.636154 8479 heartbeater.cc:503] Master 127.28.207.126:40307 was elected leader, sending a full tablet report...
Sep 29 18:34:00 dist-test-slave-qcpv krb5kdc[7789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706839, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for HTTP/127.28.207.84@KRBTEST.COM
W20260929 18:34:00.730408 8480 tablet.cc:2369] T 41653766a021454f80a515eff3e7979b P e5bc0d9b3c7a47bd949d9f98d0ca2be6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:34:00.736922 8775 raft_consensus.cc:482] T 41653766a021454f80a515eff3e7979b P 39374b51e8164b0b91d32a2d68bb97df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:34:00.737161 8775 raft_consensus.cc:504] T 41653766a021454f80a515eff3e7979b P 39374b51e8164b0b91d32a2d68bb97df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5bc0d9b3c7a47bd949d9f98d0ca2be6" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 45317 } } peers { permanent_uuid: "09fb1f3a5b7c4d64924559227b5b0e3b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37273 } } peers { permanent_uuid: "39374b51e8164b0b91d32a2d68bb97df" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46639 } }
I20260929 18:34:00.738463 8775 leader_election.cc:288] T 41653766a021454f80a515eff3e7979b P 39374b51e8164b0b91d32a2d68bb97df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5bc0d9b3c7a47bd949d9f98d0ca2be6 (127.28.207.65:45317), 09fb1f3a5b7c4d64924559227b5b0e3b (127.28.207.67:37273)
W20260929 18:34:00.747804 8750 tablet.cc:2369] T 41653766a021454f80a515eff3e7979b P 09fb1f3a5b7c4d64924559227b5b0e3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:34:00.752206 8766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:00.739361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:38607 (local address 127.28.207.67:37273)
0929 18:34:00.739562 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:00.739568 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:00.740259 (+ 691us) server_negotiation.cc:407] Connection header received
0929 18:34:00.740499 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:00.740503 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:00.740560 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:00.740609 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:34:00.741060 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:00.741852 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:00.743581 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:00.744353 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:00.744534 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:00.744606 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:00.751794 (+ 7188us) server_negotiation.cc:299] Negotiation successful
0929 18:34:00.752021 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 18:34:00.752833 8779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:00.738842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:37273 (local address 127.28.207.66:38607)
0929 18:34:00.740157 (+ 1315us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:00.740174 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:00.740349 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:00.740765 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:00.740768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:00.740786 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:34:00.740927 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:00.740934 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:00.742015 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:00.742018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:00.743380 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:00.743401 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:00.751147 (+ 7746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:00.751153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:00.751501 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:00.751549 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 18:34:00.752523 (+ 974us) client_negotiation.cc:240] Negotiation successful
0929 18:34:00.752637 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1239,"thread_start_us":116,"threads_started":1}
I20260929 18:34:00.753396 8765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:00.739954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:47549 (local address 127.28.207.65:45317)
0929 18:34:00.740080 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:00.740085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:00.746217 (+ 6132us) server_negotiation.cc:407] Connection header received
0929 18:34:00.746421 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:00.746425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:00.746478 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:00.746527 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:34:00.749100 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:00.749860 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:00.751633 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:00.752216 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:00.752352 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:00.752410 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:00.753163 (+ 753us) server_negotiation.cc:299] Negotiation successful
0929 18:34:00.753249 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 18:34:00.753724 8703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41653766a021454f80a515eff3e7979b" candidate_uuid: "39374b51e8164b0b91d32a2d68bb97df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09fb1f3a5b7c4d64924559227b5b0e3b" is_pre_election: true
I20260929 18:34:00.754179 8780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:00.739589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:45317 (local address 127.28.207.66:47549)
0929 18:34:00.746119 (+ 6530us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:00.746135 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:00.746288 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:00.748804 (+ 2516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:00.748808 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:00.748829 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:34:00.748958 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:00.748965 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:00.750404 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:00.750407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:00.751499 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:00.751508 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:00.752672 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:00.752675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:00.752787 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:00.752817 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 18:34:00.753784 (+ 967us) client_negotiation.cc:240] Negotiation successful
0929 18:34:00.753877 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6453,"thread_start_us":97,"threads_started":1}
I20260929 18:34:00.754199 8703 raft_consensus.cc:2454] T 41653766a021454f80a515eff3e7979b P 09fb1f3a5b7c4d64924559227b5b0e3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39374b51e8164b0b91d32a2d68bb97df in term 0.
I20260929 18:34:00.754927 8503 leader_election.cc:302] T 41653766a021454f80a515eff3e7979b P 39374b51e8164b0b91d32a2d68bb97df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09fb1f3a5b7c4d64924559227b5b0e3b, 39374b51e8164b0b91d32a2d68bb97df; no voters:
I20260929 18:34:00.755208 8418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41653766a021454f80a515eff3e7979b" candidate_uuid: "39374b51e8164b0b91d32a2d68bb97df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5bc0d9b3c7a47bd949d9f98d0ca2be6" is_pre_election: true
I20260929 18:34:00.755347 8775 raft_consensus.cc:2781] T 41653766a021454f80a515eff3e7979b P 39374b51e8164b0b91d32a2d68bb97df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:34:00.755425 8775 raft_consensus.cc:482] T 41653766a021454f80a515eff3e7979b P 39374b51e8164b0b91d32a2d68bb97df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:34:00.755470 8775 raft_consensus.cc:3037] T 41653766a021454f80a515eff3e7979b P 39374b51e8164b0b91d32a2d68bb97df [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:34:00.755528 8418 raft_consensus.cc:2454] T 41653766a021454f80a515eff3e7979b P e5bc0d9b3c7a47bd949d9f98d0ca2be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39374b51e8164b0b91d32a2d68bb97df in term 0.
I20260929 18:34:00.757407 8775 raft_consensus.cc:504] T 41653766a021454f80a515eff3e7979b P 39374b51e8164b0b91d32a2d68bb97df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5bc0d9b3c7a47bd949d9f98d0ca2be6" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 45317 } } peers { permanent_uuid: "09fb1f3a5b7c4d64924559227b5b0e3b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37273 } } peers { permanent_uuid: "39374b51e8164b0b91d32a2d68bb97df" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46639 } }
I20260929 18:34:00.757917 8775 leader_election.cc:288] T 41653766a021454f80a515eff3e7979b P 39374b51e8164b0b91d32a2d68bb97df [CANDIDATE]: Term 1 election: Requested vote from peers e5bc0d9b3c7a47bd949d9f98d0ca2be6 (127.28.207.65:45317), 09fb1f3a5b7c4d64924559227b5b0e3b (127.28.207.67:37273)
I20260929 18:34:00.758795 8418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41653766a021454f80a515eff3e7979b" candidate_uuid: "39374b51e8164b0b91d32a2d68bb97df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5bc0d9b3c7a47bd949d9f98d0ca2be6"
I20260929 18:34:00.758968 8418 raft_consensus.cc:3037] T 41653766a021454f80a515eff3e7979b P e5bc0d9b3c7a47bd949d9f98d0ca2be6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:34:00.760622 8418 raft_consensus.cc:2454] T 41653766a021454f80a515eff3e7979b P e5bc0d9b3c7a47bd949d9f98d0ca2be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39374b51e8164b0b91d32a2d68bb97df in term 1.
I20260929 18:34:00.761054 8505 leader_election.cc:302] T 41653766a021454f80a515eff3e7979b P 39374b51e8164b0b91d32a2d68bb97df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39374b51e8164b0b91d32a2d68bb97df, e5bc0d9b3c7a47bd949d9f98d0ca2be6; no voters:
I20260929 18:34:00.761386 8703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41653766a021454f80a515eff3e7979b" candidate_uuid: "39374b51e8164b0b91d32a2d68bb97df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09fb1f3a5b7c4d64924559227b5b0e3b"
I20260929 18:34:00.761548 8703 raft_consensus.cc:3037] T 41653766a021454f80a515eff3e7979b P 09fb1f3a5b7c4d64924559227b5b0e3b [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:34:00.762960 8775 raft_consensus.cc:2781] T 41653766a021454f80a515eff3e7979b P 39374b51e8164b0b91d32a2d68bb97df [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:34:00.763795 8703 raft_consensus.cc:2454] T 41653766a021454f80a515eff3e7979b P 09fb1f3a5b7c4d64924559227b5b0e3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39374b51e8164b0b91d32a2d68bb97df in term 1.
I20260929 18:34:00.764480 8775 raft_consensus.cc:686] T 41653766a021454f80a515eff3e7979b P 39374b51e8164b0b91d32a2d68bb97df [term 1 LEADER]: Becoming Leader. State: Replica: 39374b51e8164b0b91d32a2d68bb97df, State: Running, Role: LEADER
I20260929 18:34:00.765007 8775 consensus_queue.cc:237] T 41653766a021454f80a515eff3e7979b P 39374b51e8164b0b91d32a2d68bb97df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5bc0d9b3c7a47bd949d9f98d0ca2be6" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 45317 } } peers { permanent_uuid: "09fb1f3a5b7c4d64924559227b5b0e3b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37273 } } peers { permanent_uuid: "39374b51e8164b0b91d32a2d68bb97df" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46639 } }
I20260929 18:34:00.769623 8260 catalog_manager.cc:5495] T 41653766a021454f80a515eff3e7979b P 39374b51e8164b0b91d32a2d68bb97df reported cstate change: term changed from 0 to 1, leader changed from <none> to 39374b51e8164b0b91d32a2d68bb97df (127.28.207.66). New cstate: current_term: 1 leader_uuid: "39374b51e8164b0b91d32a2d68bb97df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5bc0d9b3c7a47bd949d9f98d0ca2be6" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 45317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09fb1f3a5b7c4d64924559227b5b0e3b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39374b51e8164b0b91d32a2d68bb97df" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46639 } health_report { overall_health: HEALTHY } } }
W20260929 18:34:00.801770 8616 tablet.cc:2369] T 41653766a021454f80a515eff3e7979b P 39374b51e8164b0b91d32a2d68bb97df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:34:00.830823 8767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:00.824374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39322 (local address 127.28.207.66:46639)
0929 18:34:00.824522 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:00.824528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:00.824547 (+ 19us) server_negotiation.cc:407] Connection header received
0929 18:34:00.824734 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:00.824738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:00.824822 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:00.824951 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:34:00.825513 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:00.826461 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:00.827998 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:00.828395 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:00.828509 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:00.829024 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:34:00.829911 (+ 887us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:34:00.830054 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:00.830572 (+ 518us) server_negotiation.cc:299] Negotiation successful
0929 18:34:00.830647 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 18:34:00.840477 8703 raft_consensus.cc:1261] T 41653766a021454f80a515eff3e7979b P 09fb1f3a5b7c4d64924559227b5b0e3b [term 1 FOLLOWER]: Refusing update from remote peer 39374b51e8164b0b91d32a2d68bb97df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:34:00.841539 8418 raft_consensus.cc:1261] T 41653766a021454f80a515eff3e7979b P e5bc0d9b3c7a47bd949d9f98d0ca2be6 [term 1 FOLLOWER]: Refusing update from remote peer 39374b51e8164b0b91d32a2d68bb97df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:34:00.842653 8775 consensus_queue.cc:1035] T 41653766a021454f80a515eff3e7979b P 39374b51e8164b0b91d32a2d68bb97df [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5bc0d9b3c7a47bd949d9f98d0ca2be6" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:34:00.843768 8775 consensus_queue.cc:1035] T 41653766a021454f80a515eff3e7979b P 39374b51e8164b0b91d32a2d68bb97df [LEADER]: Connected to new peer: Peer: permanent_uuid: "09fb1f3a5b7c4d64924559227b5b0e3b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 37273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:34:00.855906 8784 mvcc.cc:205] Tried to move back new op lower bound from 7334735220068818944 to 7334735219779080192. Current Snapshot: MvccSnapshot[applied={T|T < 7334735220068818944}]
I20260929 18:34:00.860010 8786 mvcc.cc:205] Tried to move back new op lower bound from 7334735220068818944 to 7334735219779080192. Current Snapshot: MvccSnapshot[applied={T|T < 7334735220068818944}]
I20260929 18:34:00.867166 8785 mvcc.cc:205] Tried to move back new op lower bound from 7334735220068818944 to 7334735219779080192. Current Snapshot: MvccSnapshot[applied={T|T < 7334735220068818944}]
I20260929 18:34:00.870810 8260 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:34:00.871135 8260 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:34:00.874548 8260 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 080b9911896046dbb63c5f099ca192d0: Sending DeleteTablet for 3 replicas of tablet 41653766a021454f80a515eff3e7979b
I20260929 18:34:00.876703 8413 tablet_service.cc:1514] Processing DeleteTablet for tablet 41653766a021454f80a515eff3e7979b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:34:00 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:48868
I20260929 18:34:00.877669 8683 tablet_service.cc:1514] Processing DeleteTablet for tablet 41653766a021454f80a515eff3e7979b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:34:00 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:38916
I20260929 18:34:00.878229 8533 tablet_service.cc:1514] Processing DeleteTablet for tablet 41653766a021454f80a515eff3e7979b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:34:00 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:39308
I20260929 18:34:00.879463 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 8346
I20260929 18:34:00.887181 8795 tablet_replica.cc:330] T 41653766a021454f80a515eff3e7979b P 39374b51e8164b0b91d32a2d68bb97df: stopping tablet replica
I20260929 18:34:00.887699 8795 raft_consensus.cc:2229] T 41653766a021454f80a515eff3e7979b P 39374b51e8164b0b91d32a2d68bb97df [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:34:00.888329 8795 raft_consensus.cc:2258] T 41653766a021454f80a515eff3e7979b P 39374b51e8164b0b91d32a2d68bb97df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:34:00.891126 8794 tablet_replica.cc:330] T 41653766a021454f80a515eff3e7979b P 09fb1f3a5b7c4d64924559227b5b0e3b: stopping tablet replica
I20260929 18:34:00.891516 8794 raft_consensus.cc:2229] T 41653766a021454f80a515eff3e7979b P 09fb1f3a5b7c4d64924559227b5b0e3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:34:00.891911 8794 raft_consensus.cc:2258] T 41653766a021454f80a515eff3e7979b P 09fb1f3a5b7c4d64924559227b5b0e3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:34:00.893941 8794 ts_tablet_manager.cc:1905] T 41653766a021454f80a515eff3e7979b P 09fb1f3a5b7c4d64924559227b5b0e3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:34:00.897332 8794 ts_tablet_manager.cc:1918] T 41653766a021454f80a515eff3e7979b P 09fb1f3a5b7c4d64924559227b5b0e3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:34:00.897444 8794 log.cc:1196] T 41653766a021454f80a515eff3e7979b P 09fb1f3a5b7c4d64924559227b5b0e3b: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-2/wal/wals/41653766a021454f80a515eff3e7979b
I20260929 18:34:00.897796 8794 ts_tablet_manager.cc:1939] T 41653766a021454f80a515eff3e7979b P 09fb1f3a5b7c4d64924559227b5b0e3b: Deleting consensus metadata
I20260929 18:34:00.900429 8795 ts_tablet_manager.cc:1905] T 41653766a021454f80a515eff3e7979b P 39374b51e8164b0b91d32a2d68bb97df: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 18:34:00.903533 8252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.207.65:45317 (error 108)
I20260929 18:34:00.903765 8795 ts_tablet_manager.cc:1918] T 41653766a021454f80a515eff3e7979b P 39374b51e8164b0b91d32a2d68bb97df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:34:00.903924 8795 log.cc:1196] T 41653766a021454f80a515eff3e7979b P 39374b51e8164b0b91d32a2d68bb97df: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790706718260899-29501-0/minicluster-data/ts-1/wal/wals/41653766a021454f80a515eff3e7979b
I20260929 18:34:00.904285 8795 ts_tablet_manager.cc:1939] T 41653766a021454f80a515eff3e7979b P 39374b51e8164b0b91d32a2d68bb97df: Deleting consensus metadata
I20260929 18:34:00.905213 8250 catalog_manager.cc:4841] TS 09fb1f3a5b7c4d64924559227b5b0e3b (127.28.207.67:37273): tablet 41653766a021454f80a515eff3e7979b (table test-table [id=9a9a6be5d7be4a4f9892c8159cd7b6ea]) successfully deleted
I20260929 18:34:00.905699 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 8485
I20260929 18:34:00.905790 8250 catalog_manager.cc:4841] TS 39374b51e8164b0b91d32a2d68bb97df (127.28.207.66:46639): tablet 41653766a021454f80a515eff3e7979b (table test-table [id=9a9a6be5d7be4a4f9892c8159cd7b6ea]) successfully deleted
I20260929 18:34:00.908386 8763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:00.908016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:45317 (local address 127.0.0.1:48872)
0929 18:34:00.908180 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:00.908267 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.65:45317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260929 18:34:00.915026 8252 catalog_manager.cc:4584] TS e5bc0d9b3c7a47bd949d9f98d0ca2be6 (127.28.207.65:45317): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 41653766a021454f80a515eff3e7979b: Network error: Client connection negotiation failed: client connection to 127.28.207.65:45317: connect: Connection refused (error 111)
I20260929 18:34:00.915840 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 8620
I20260929 18:34:00.923381 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 8233
I20260929 18:34:00.931465 29501 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 18:34:01.277709 29501 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 18:34:01.277951 29501 mini_ranger.cc:66] Stopping Ranger...
I20260929 18:34:01.623845 29501 mini_ranger.cc:68] Stopped Ranger
2026-09-29 18:34:01.624 UTC [7819] LOG: received smart shutdown request
2026-09-29 18:34:01.628 UTC [7819] LOG: background worker "logical replication launcher" (PID 7828) exited with exit code 1
2026-09-29 18:34:01.628 UTC [7823] LOG: shutting down
2026-09-29 18:34:01.827 UTC [7819] LOG: database system is shut down
2026-09-29T18:34:01Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (55916 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:34:02 dist-test-slave-qcpv krb5kdc[8805](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:34:02 dist-test-slave-qcpv krb5kdc[8805](info): set up 2 sockets
Sep 29 18:34:02 dist-test-slave-qcpv krb5kdc[8805](info): commencing operation
krb5kdc: starting...
W20260929 18:34:03.027207 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:34:03 dist-test-slave-qcpv krb5kdc[8805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:34:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:34:03Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.28.207.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.28.207.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.28.207.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/rangeradmin_127.28.207.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.28.207.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/rangeradmin_127.28.207.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.28.207.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.28.207.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.28.207.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/rangerlookup_127.28.207.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.28.207.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/rangerlookup_127.28.207.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.207.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.28.207.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.207.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/HTTP_127.28.207.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.207.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/HTTP_127.28.207.84@KRBTEST.COM.keytab.
I20260929 18:34:03.222745 29501 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskxlLuiw/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/postgres -l logfile start
2026-09-29 18:34:06.273 UTC [8835] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 18:34:06.274 UTC [8835] LOG: listening on IPv4 address "127.28.207.84", port 57643
2026-09-29 18:34:06.279 UTC [8835] LOG: listening on Unix socket "/tmp/.s.PGSQL.57643"
2026-09-29 18:34:06.304 UTC [8838] LOG: database system was shut down at 2026-09-29 18:34:04 UTC
2026-09-29 18:34:06.308 UTC [8835] LOG: database system is ready to accept connections
I20260929 18:34:07.270795 29501 mini_postgres.cc:96] Postgres bound to 57643
2026-09-29 18:34:07.278 UTC [8846] FATAL: database "slave" does not exist
127.28.207.84:57643 - accepting connections
I20260929 18:34:07.279910 29501 mini_ranger.cc:161] Starting Ranger...
I20260929 18:34:07.300431 29501 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 18:34:07.925659 29501 mini_ranger.cc:87] Created ranger Postgres database
I20260929 18:34:07.925756 29501 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ranger-admin
2026-09-29 18:34:08,225 [I] DB FLAVOR :POSTGRES
2026-09-29 18:34:08,225 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 18:34:08,226 [I] Checking connection..
2026-09-29 18:34:08,226 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:57643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 18:34:08,492 [I] Checking connection passed.
2026-09-29 18:34:08,492 [I] --------- Verifying version history table ---------
2026-09-29 18:34:08,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:57643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 18:34:08,768 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 18:34:08,768 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:57643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 18:34:09,063 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 18:34:09,063 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 18:34:09,063 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:57643/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 18:34:09.360 UTC [8910] WARNING: there is no transaction in progress
2026-09-29 18:34:09,368 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 18:34:09,368 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:57643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 18:34:09,634 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 18:34:09,634 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 18:34:09,635 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:57643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 18:34:09,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:57643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 18:34:10,180 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:57643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-qcpv.c.gcp-upstream.internal','N') ;"
2026-09-29 18:34:10,484 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:57643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 18:34:10,785 [I] Table x_portal_user does not exist in database ranger
2026-09-29 18:34:10,785 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:57643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 18:34:11,065 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 18:34:11,065 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:57643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 18:34:11,342 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:57643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 18:34:11,597 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 18:34:11,598 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:57643/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 18:34:12.781 UTC [9018] WARNING: there is no transaction in progress
2026-09-29 18:34:12.801 UTC [9018] WARNING: there is no transaction in progress
2026-09-29 18:34:12.823 UTC [9018] WARNING: there is no transaction in progress
2026-09-29 18:34:12.843 UTC [9018] WARNING: there is no transaction in progress
2026-09-29 18:34:12.861 UTC [9018] WARNING: there is no transaction in progress
2026-09-29 18:34:12.886 UTC [9018] WARNING: there is no transaction in progress
2026-09-29 18:34:12.947 UTC [9018] WARNING: there is no transaction in progress
2026-09-29 18:34:12.955 UTC [9018] WARNING: there is no transaction in progress
2026-09-29 18:34:12.972 UTC [9018] WARNING: there is no transaction in progress
2026-09-29 18:34:12.983 UTC [9018] WARNING: there is no transaction in progress
2026-09-29 18:34:12.995 UTC [9018] WARNING: there is no transaction in progress
2026-09-29 18:34:13.011 UTC [9018] WARNING: there is no transaction in progress
2026-09-29 18:34:13.023 UTC [9018] WARNING: there is no transaction in progress
2026-09-29 18:34:13.031 UTC [9018] WARNING: there is no transaction in progress
2026-09-29 18:34:13.039 UTC [9018] WARNING: there is no transaction in progress
2026-09-29 18:34:13.047 UTC [9018] WARNING: there is no transaction in progress
2026-09-29 18:34:13,862 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 18:34:13,863 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:57643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 18:34:14,165 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 18:34:14,166 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:57643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 18:34:14,469 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 18:34:14,470 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:57643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 18:34:14,749 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 18:34:14,750 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:57643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 18:34:15,049 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:57643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 18:34:15,294 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:57643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-qcpv.c.gcp-upstream.internal';"
2026-09-29 18:34:15,578 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 18:34:15,579 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:57643/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 18:34:15,876 [I] DB_PATCHES have already been applied
I20260929 18:34:15.882951 29501 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ranger-admin:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskxlLuiw/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 18:34:15.883059 29501 mini_ranger.cc:193] Using host: 127.28.207.84
I20260929 18:34:15.887501 29501 mini_ranger.cc:227] Ranger admin URL: http://127.28.207.84:42189
Sep 29, 2026 6:34:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 6:34:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 6:34:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 6:34:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 6:34:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 6:34:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 6:34:16 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 18:34:17.041123 29501 mini_ranger.cc:160] Time spent starting Ranger: real 9.761s user 0.009s sys 0.004s
Sep 29, 2026 6:34:40 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 6:34:40 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 29, 2026 6:34:40 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 6:34:40 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 6:34:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 6:34:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 6:34:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 6:34:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 6:34:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 6:34:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 6:34:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 6:34:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 6:34:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 6:34:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 6:34:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 6:34:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 6:34:40 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 6:34:41 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 18:34:42.616314 29501 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.28.207.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.28.207.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.28.207.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/rangerkms_127.28.207.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.28.207.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/rangerkms_127.28.207.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.28.207.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.28.207.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.28.207.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/HTTP_127.28.207.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.28.207.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/HTTP_127.28.207.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 18:34:42 dist-test-slave-qcpv krb5kdc[8805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706882, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 18:34:42.706722 29501 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 18:34:42.717831 29501 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 18:34:43.805297 29501 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 18:34:43.805398 29501 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ranger-kms
I20260929 18:34:43.805419 29501 mini_ranger_kms.cc:227] Using postgres at 127.28.207.84:57643
2026-09-29 18:34:43,995 [I] DB FLAVOR :POSTGRES
2026-09-29 18:34:43,996 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 18:34:43,996 [I] Checking connection
2026-09-29 18:34:43,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:57643/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 18:34:44,292 [I] connection success
2026-09-29 18:34:44,292 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 18:34:44,292 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 18:34:44,292 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:57643/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 18:34:44,534 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 18:34:44,535 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 18:34:44,535 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 18:34:44,535 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.28.207.84:57643/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 18:34:44,812 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 18:34:45.400236 29501 mini_ranger_kms.cc:310] Created kms service
I20260929 18:34:45.629249 29501 mini_ranger_kms.cc:326] Created kudu user
I20260929 18:34:45.685046 29501 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 18:34:45 dist-test-slave-qcpv krb5kdc[8805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706882, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.28.207.84@KRBTEST.COM
I20260929 18:34:46.184001 29501 mini_ranger_kms.cc:384] Added ranger policy
I20260929 18:34:46.184160 29501 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ranger-kms:/tmp/dist-test-taskxlLuiw/build/debug/bin/postgresql.jar:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskxlLuiw/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 18:34:46.184181 29501 mini_ranger_kms.cc:240] Using host: 127.28.207.84
I20260929 18:34:46.189235 29501 mini_ranger_kms.cc:276] Ranger KMS PID: 9199
I20260929 18:34:46.189370 29501 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.28.207.84:42421
Sep 29, 2026 6:34:47 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 6:34:47 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 6:34:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 6:34:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 6:34:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 6:34:47 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 18:34:48.476395 29501 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.770s user 0.013s sys 0.153s
I20260929 18:34:48.476553 29501 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 18:34:48.476637 29501 mini_ranger_kms.cc:401] 127.28.207.84:42421/kms/v1/keys
Sep 29 18:34:48 dist-test-slave-qcpv krb5kdc[8805](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790706888, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.207.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 18:34:49 dist-test-slave-qcpv krb5kdc[8805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706888, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.207.84@KRBTEST.COM for HTTP/127.28.207.84@KRBTEST.COM
Sep 29 18:34:49 dist-test-slave-qcpv krb5kdc[8805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706888, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.28.207.84@KRBTEST.COM for HTTP/127.28.207.84@KRBTEST.COM
Sep 29 18:34:52 dist-test-slave-qcpv krb5kdc[8805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.28.207.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
I20260929 18:34:52.276031 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:34973
--webserver_interface=127.28.207.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:37075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:34973
--ranger_config_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.207.84:42421/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:34:52.429330 9249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:34:52.429638 9249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:34:52.429682 9249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:34:52.434399 9249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:34:52.434478 9249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:34:52.434500 9249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:34:52.434518 9249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:34:52.434536 9249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:34:52.440447 9249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:37075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/master-0/wal
--ranger_kms_url=127.28.207.84:42421/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:34973
--ranger_config_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:34973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:34:52.441593 9249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:34:52.442682 9249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:34:52.451931 9254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:34:52.452157 9249 server_base.cc:1034] running on GCE node
W20260929 18:34:52.451934 9255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:52.452512 9257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:34:52.453186 9249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:34:52.454418 9249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:34:52.455600 9249 hybrid_clock.cc:648] HybridClock initialized: now 1790706892455565 us; error 57 us; skew 500 ppm
Sep 29 18:34:52 dist-test-slave-qcpv krb5kdc[8805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706892, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:34:52.458950 9249 init.cc:377] Logged in from keytab as kudu/127.28.207.126@KRBTEST.COM (short username kudu)
I20260929 18:34:52.460022 9249 webserver.cc:460] Webserver started at http://127.28.207.126:37279/ using document root <none> and password file <none>
I20260929 18:34:52.460599 9249 fs_manager.cc:359] Metadata directory not provided
I20260929 18:34:52.460682 9249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:34:52.460898 9249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:34:52 dist-test-slave-qcpv krb5kdc[8805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706892, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for HTTP/127.28.207.84@KRBTEST.COM
I20260929 18:34:52.514310 9249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "cf0a84c3350545f6966cc2b9b3c3dac0"
format_stamp: "Formatted at 2026-09-29 18:34:52 on dist-test-slave-qcpv"
server_key: "b7c140968fb70ed2858701de29bc51f0"
server_key_iv: "50890ccfcb91d179841578f4bce39b23"
server_key_version: "kuduclusterkey@0"
I20260929 18:34:52.515107 9249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "cf0a84c3350545f6966cc2b9b3c3dac0"
format_stamp: "Formatted at 2026-09-29 18:34:52 on dist-test-slave-qcpv"
server_key: "b7c140968fb70ed2858701de29bc51f0"
server_key_iv: "50890ccfcb91d179841578f4bce39b23"
server_key_version: "kuduclusterkey@0"
I20260929 18:34:52.519500 9249 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:34:52.535145 9264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:52.536777 9249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 18:34:52.537006 9249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "cf0a84c3350545f6966cc2b9b3c3dac0"
format_stamp: "Formatted at 2026-09-29 18:34:52 on dist-test-slave-qcpv"
server_key: "b7c140968fb70ed2858701de29bc51f0"
server_key_iv: "50890ccfcb91d179841578f4bce39b23"
server_key_version: "kuduclusterkey@0"
I20260929 18:34:52.537207 9249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:34:52.555802 9249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:34:52.560030 9249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:34:52.560592 9249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:34:52.569608 9249 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:34973
I20260929 18:34:52.569695 9316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:34973 every 8 connection(s)
I20260929 18:34:52.570962 9249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:34:52.575273 9317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:34:52.577153 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 9249
I20260929 18:34:52.577315 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:34:52.577697 29501 external_mini_cluster.cc:1428] Setting key 9deb6abca59d24f8afad2bf403967bda
I20260929 18:34:52.584122 9317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0: Bootstrap starting.
I20260929 18:34:52.587426 9317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0: Neither blocks nor log segments found. Creating new log.
I20260929 18:34:52.588629 9317 log.cc:824] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:34:52 dist-test-slave-qcpv krb5kdc[8805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:34:52.591781 9317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0: No bootstrap required, opened a new log
I20260929 18:34:52.596498 9317 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf0a84c3350545f6966cc2b9b3c3dac0" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 34973 } }
I20260929 18:34:52.596817 9317 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:34:52.596880 9317 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf0a84c3350545f6966cc2b9b3c3dac0, State: Initialized, Role: FOLLOWER
I20260929 18:34:52.597468 9317 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf0a84c3350545f6966cc2b9b3c3dac0" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 34973 } }
I20260929 18:34:52.597602 9317 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:34:52.597656 9320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:52.579649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41750 (local address 127.28.207.126:34973)
0929 18:34:52.580318 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:52.580342 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:52.580402 (+ 60us) server_negotiation.cc:407] Connection header received
0929 18:34:52.581546 (+ 1144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:52.581576 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:52.582115 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:52.582521 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:34:52.583782 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:52.585396 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:52.586773 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:52.587581 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:52.587815 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:52.591175 (+ 3360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:34:52.591183 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:34:52.591205 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:34:52.591242 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:34:52.593996 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:52.594681 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:52.594687 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:52.594692 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:52.594819 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:52.595267 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:52.595270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:52.595271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:52.595729 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:34:52.595921 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:52.596245 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 18:34:52.596583 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":195,"threads_started":1}
I20260929 18:34:52.597699 9317 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:34:52.597852 9317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:34:52.599097 9317 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf0a84c3350545f6966cc2b9b3c3dac0" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 34973 } }
I20260929 18:34:52.599520 9317 leader_election.cc:302] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf0a84c3350545f6966cc2b9b3c3dac0; no voters:
I20260929 18:34:52.599969 9317 leader_election.cc:288] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:34:52.600087 9322 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:34:52.600399 9322 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0 [term 1 LEADER]: Becoming Leader. State: Replica: cf0a84c3350545f6966cc2b9b3c3dac0, State: Running, Role: LEADER
I20260929 18:34:52.600870 9322 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf0a84c3350545f6966cc2b9b3c3dac0" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 34973 } }
I20260929 18:34:52.601615 9317 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:34:52.602828 9323 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf0a84c3350545f6966cc2b9b3c3dac0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf0a84c3350545f6966cc2b9b3c3dac0" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 34973 } } }
I20260929 18:34:52.603020 9323 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0 [sys.catalog]: This master's current role is: LEADER
I20260929 18:34:52.603446 9331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:34:52.604215 9324 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf0a84c3350545f6966cc2b9b3c3dac0. Latest consensus state: current_term: 1 leader_uuid: "cf0a84c3350545f6966cc2b9b3c3dac0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf0a84c3350545f6966cc2b9b3c3dac0" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 34973 } } }
I20260929 18:34:52.604336 9324 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0 [sys.catalog]: This master's current role is: LEADER
I20260929 18:34:52.606272 9331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:34:52.606824 9317 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 18:34:52.617308 9331 catalog_manager.cc:1347] Generated new cluster ID: b0c97a7a9286480283f08847528e73d1
I20260929 18:34:52.617409 9331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:34:52.631446 9331 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:34:52.633145 9331 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:34:52.644112 9331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0: Generated new TSK 0
I20260929 18:34:52.645185 9331 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.207.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab.
Entry for principal kudu/127.28.207.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab.
I20260929 18:34:53.426985 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.207.84:42421/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.65:0
--local_ip_for_outbound_sockets=127.28.207.65
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:34973
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:37075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:34:53.609736 9362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:34:53.610594 9362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:34:53.610742 9362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:34:53.618649 9362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:34:53.618796 9362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:34:53.618937 9362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.65
I20260929 18:34:53.628615 9362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:37075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.28.207.84:42421/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:34973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.65
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:34:53.630649 9362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:34:53.632512 9362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:34:53.648784 9370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:53.648816 9369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:53.654685 9372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:34:53.656029 9362 server_base.cc:1034] running on GCE node
I20260929 18:34:53.656737 9362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:34:53.657750 9362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:34:53.659096 9362 hybrid_clock.cc:648] HybridClock initialized: now 1790706893659046 us; error 63 us; skew 500 ppm
Sep 29 18:34:53 dist-test-slave-qcpv krb5kdc[8805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706893, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:34:53.663398 9362 init.cc:377] Logged in from keytab as kudu/127.28.207.65@KRBTEST.COM (short username kudu)
I20260929 18:34:53.664909 9362 webserver.cc:460] Webserver started at http://127.28.207.65:39253/ using document root <none> and password file <none>
I20260929 18:34:53.665858 9362 fs_manager.cc:359] Metadata directory not provided
I20260929 18:34:53.666020 9362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:34:53.666358 9362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:34:53 dist-test-slave-qcpv krb5kdc[8805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706893, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for HTTP/127.28.207.84@KRBTEST.COM
I20260929 18:34:53.693984 9362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-0/data/instance:
uuid: "a337656f4c7340cca6c6a620452e40d1"
format_stamp: "Formatted at 2026-09-29 18:34:53 on dist-test-slave-qcpv"
server_key: "dbec8b250afec688d70319697804200e"
server_key_iv: "c2676566bfa6ef8cb21ae4b1b7dc3f58"
server_key_version: "kuduclusterkey@0"
I20260929 18:34:53.694708 9362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance:
uuid: "a337656f4c7340cca6c6a620452e40d1"
format_stamp: "Formatted at 2026-09-29 18:34:53 on dist-test-slave-qcpv"
server_key: "dbec8b250afec688d70319697804200e"
server_key_iv: "c2676566bfa6ef8cb21ae4b1b7dc3f58"
server_key_version: "kuduclusterkey@0"
I20260929 18:34:53.716037 9362 fs_manager.cc:691] Time spent creating directory manager: real 0.021s user 0.000s sys 0.004s
I20260929 18:34:53.745244 9381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:53.748553 9362 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.003s sys 0.001s
I20260929 18:34:53.748740 9362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-0/wal
uuid: "a337656f4c7340cca6c6a620452e40d1"
format_stamp: "Formatted at 2026-09-29 18:34:53 on dist-test-slave-qcpv"
server_key: "dbec8b250afec688d70319697804200e"
server_key_iv: "c2676566bfa6ef8cb21ae4b1b7dc3f58"
server_key_version: "kuduclusterkey@0"
I20260929 18:34:53.748914 9362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:34:53.791225 9362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
Sep 29 18:34:53 dist-test-slave-qcpv krb5kdc[8805](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790706893, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:34:53.794462 9362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:34:53.794685 9362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:34:53.795449 9362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:34:53.796409 9362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:34:53.796476 9362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:53.796541 9362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:34:53.796566 9362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:53.807719 9362 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.65:42631
I20260929 18:34:53.809028 9362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
I20260929 18:34:53.809768 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 9362
I20260929 18:34:53.809885 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance
I20260929 18:34:53.810221 29501 external_mini_cluster.cc:1428] Setting key f1c6a10f20d4eca2fd293343522e0a24
I20260929 18:34:53.820406 9494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.65:42631 every 8 connection(s)
WARNING: no policy specified for kudu/127.28.207.66@KRBTEST.COM; defaulting to no policy
Sep 29 18:34:53 dist-test-slave-qcpv krb5kdc[8805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706893, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.66@KRBTEST.COM" created.
I20260929 18:34:53.842263 9499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:53.824912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:44979 (local address 127.28.207.126:34973)
0929 18:34:53.825266 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:53.825273 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:53.827359 (+ 2086us) server_negotiation.cc:407] Connection header received
0929 18:34:53.827423 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:53.827429 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:53.827503 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:53.827601 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:34:53.829409 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:53.830318 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:53.832210 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:53.832456 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:53.832579 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:53.836085 (+ 3506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:34:53.836093 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:34:53.836099 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:34:53.836146 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:34:53.838297 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:53.839167 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:53.839173 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:53.839177 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:53.839287 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:53.839734 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:53.839740 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:53.839744 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:53.840029 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:34:53.840126 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:53.841867 (+ 1741us) server_negotiation.cc:299] Negotiation successful
0929 18:34:53.842028 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":83,"threads_started":1}
I20260929 18:34:53.843410 9498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:53.823741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:34973 (local address 127.28.207.65:44979)
0929 18:34:53.825597 (+ 1856us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:53.825660 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:53.827003 (+ 1343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:53.827988 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:53.828003 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:53.828420 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:34:53.829197 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:53.829222 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:53.830492 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:53.830495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:53.832060 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:53.832075 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:53.832627 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:53.832632 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:53.832861 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:53.833523 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:34:53.833540 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:34:53.835800 (+ 2260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:34:53.838535 (+ 2735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:34:53.838543 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:34:53.838568 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:34:53.839033 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:34:53.839411 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:34:53.839416 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:34:53.839420 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:34:53.839622 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:34:53.840184 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:34:53.840195 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:34:53.840578 (+ 383us) client_negotiation.cc:769] Sending connection context
0929 18:34:53.840809 (+ 231us) client_negotiation.cc:240] Negotiation successful
0929 18:34:53.841223 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1594,"thread_start_us":188,"threads_started":1}
I20260929 18:34:53.845096 9495 heartbeater.cc:347] Connected to a master server at 127.28.207.126:34973
I20260929 18:34:53.845360 9495 heartbeater.cc:464] Registering TS with master...
I20260929 18:34:53.846004 9495 heartbeater.cc:511] Master 127.28.207.126:34973 requested a full tablet report, sending...
I20260929 18:34:53.847882 9281 ts_manager.cc:194] Registered new tserver with Master: a337656f4c7340cca6c6a620452e40d1 (127.28.207.65:42631)
I20260929 18:34:53.849197 9281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.65@KRBTEST.COM'} at 127.28.207.65:44979
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab.
Entry for principal kudu/127.28.207.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab.
I20260929 18:34:53.859295 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.207.84:42421/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.66:0
--local_ip_for_outbound_sockets=127.28.207.66
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:34973
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:37075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:34:54.020445 9501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:34:54.020924 9501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:34:54.021049 9501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:34:54.026204 9501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:34:54.026307 9501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:34:54.026439 9501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.66
I20260929 18:34:54.033900 9501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:37075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.28.207.84:42421/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:34973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.66
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:34:54.035614 9501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:34:54.037056 9501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:34:54.050204 9510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:54.050122 9508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:54.050609 9507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:34:54.051028 9501 server_base.cc:1034] running on GCE node
I20260929 18:34:54.051596 9501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:34:54.052345 9501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:34:54.053573 9501 hybrid_clock.cc:648] HybridClock initialized: now 1790706894053536 us; error 50 us; skew 500 ppm
Sep 29 18:34:54 dist-test-slave-qcpv krb5kdc[8805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706894, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:34:54.057356 9501 init.cc:377] Logged in from keytab as kudu/127.28.207.66@KRBTEST.COM (short username kudu)
I20260929 18:34:54.058662 9501 webserver.cc:460] Webserver started at http://127.28.207.66:44809/ using document root <none> and password file <none>
I20260929 18:34:54.059449 9501 fs_manager.cc:359] Metadata directory not provided
I20260929 18:34:54.059549 9501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:34:54.059815 9501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:34:54 dist-test-slave-qcpv krb5kdc[8805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706894, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for HTTP/127.28.207.84@KRBTEST.COM
I20260929 18:34:54.094017 9501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-1/data/instance:
uuid: "2004d94307344d29bda62f6cd1500167"
format_stamp: "Formatted at 2026-09-29 18:34:54 on dist-test-slave-qcpv"
server_key: "7dce3e9da78286745e4f5b6f90eb04d7"
server_key_iv: "63311e5f90ecf6ee45752f0c926925d0"
server_key_version: "kuduclusterkey@0"
I20260929 18:34:54.094933 9501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance:
uuid: "2004d94307344d29bda62f6cd1500167"
format_stamp: "Formatted at 2026-09-29 18:34:54 on dist-test-slave-qcpv"
server_key: "7dce3e9da78286745e4f5b6f90eb04d7"
server_key_iv: "63311e5f90ecf6ee45752f0c926925d0"
server_key_version: "kuduclusterkey@0"
I20260929 18:34:54.102974 9501 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.000s
I20260929 18:34:54.122406 9517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:54.124560 9501 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:34:54.124743 9501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-1/wal
uuid: "2004d94307344d29bda62f6cd1500167"
format_stamp: "Formatted at 2026-09-29 18:34:54 on dist-test-slave-qcpv"
server_key: "7dce3e9da78286745e4f5b6f90eb04d7"
server_key_iv: "63311e5f90ecf6ee45752f0c926925d0"
server_key_version: "kuduclusterkey@0"
I20260929 18:34:54.124917 9501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:34:54.160084 9501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:34:54.174098 9501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:34:54.174316 9501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:34:54.175123 9501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:34:54.176218 9501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:34:54.176287 9501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:54.176352 9501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:34:54.176388 9501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:54.189476 9501 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.66:44127
I20260929 18:34:54.190665 9501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
I20260929 18:34:54.190927 9630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.66:44127 every 8 connection(s)
I20260929 18:34:54.195124 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 9501
I20260929 18:34:54.195279 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance
I20260929 18:34:54.195636 29501 external_mini_cluster.cc:1428] Setting key 57e414b78da8ac5e74657145bac12efd
Sep 29 18:34:54 dist-test-slave-qcpv krb5kdc[8805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706894, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:34:54.212262 9499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:54.193620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:33987 (local address 127.28.207.126:34973)
0929 18:34:54.193757 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:54.193762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:54.194757 (+ 995us) server_negotiation.cc:407] Connection header received
0929 18:34:54.196025 (+ 1268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:54.196029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:54.196087 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:54.196143 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:34:54.197821 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:54.198666 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:54.200398 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:54.200762 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:54.200850 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:54.205069 (+ 4219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:34:54.205074 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:34:54.205077 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:34:54.205113 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:34:54.207085 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:54.210077 (+ 2992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:54.210081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:54.210083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:54.210186 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:54.210662 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:54.210665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:54.210667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:54.210956 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:34:54.211058 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:54.211888 (+ 830us) server_negotiation.cc:299] Negotiation successful
0929 18:34:54.212053 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:34:54.214001 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:54.194083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:34973 (local address 127.28.207.66:33987)
0929 18:34:54.194600 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:54.194653 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:54.195759 (+ 1106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:54.196571 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:54.196580 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:54.196953 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:34:54.197618 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:54.197639 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:54.198831 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:54.198834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:54.200250 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:54.200261 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:54.201067 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:54.201071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:54.201274 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:54.202028 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:34:54.202050 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:34:54.204754 (+ 2704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:34:54.209401 (+ 4647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:34:54.209408 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:34:54.209426 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:34:54.209904 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:34:54.210319 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:34:54.210321 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:34:54.210323 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:34:54.210532 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:34:54.211131 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:34:54.211138 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:34:54.211574 (+ 436us) client_negotiation.cc:769] Sending connection context
0929 18:34:54.212485 (+ 911us) client_negotiation.cc:240] Negotiation successful
0929 18:34:54.212881 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":182,"threads_started":1}
I20260929 18:34:54.215595 9631 heartbeater.cc:347] Connected to a master server at 127.28.207.126:34973
I20260929 18:34:54.215855 9631 heartbeater.cc:464] Registering TS with master...
I20260929 18:34:54.216476 9631 heartbeater.cc:511] Master 127.28.207.126:34973 requested a full tablet report, sending...
I20260929 18:34:54.217661 9281 ts_manager.cc:194] Registered new tserver with Master: 2004d94307344d29bda62f6cd1500167 (127.28.207.66:44127)
I20260929 18:34:54.218429 9281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.66@KRBTEST.COM'} at 127.28.207.66:33987
WARNING: no policy specified for kudu/127.28.207.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab.
Entry for principal kudu/127.28.207.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab.
I20260929 18:34:54.248829 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.207.84:42421/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.67:0
--local_ip_for_outbound_sockets=127.28.207.67
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:34973
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:37075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:34:54 dist-test-slave-qcpv krb5kdc[8805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706893, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for HTTP/127.28.207.84@KRBTEST.COM
W20260929 18:34:54.454705 9636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:34:54.455790 9636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:34:54.455878 9636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:34:54.463912 9636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:34:54.464056 9636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:34:54.464280 9636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.67
I20260929 18:34:54.475118 9636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:37075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.28.207.84:42421/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:34973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.67
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:34:54.476830 9636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:34:54.478226 9636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:34:54.493549 9642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:54.494048 9644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:34:54.494112 9636 server_base.cc:1034] running on GCE node
W20260929 18:34:54.495045 9641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:34:54.495648 9636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:34:54.496573 9636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:34:54.497818 9636 hybrid_clock.cc:648] HybridClock initialized: now 1790706894497773 us; error 50 us; skew 500 ppm
Sep 29 18:34:54 dist-test-slave-qcpv krb5kdc[8805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706894, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:34:54.501936 9636 init.cc:377] Logged in from keytab as kudu/127.28.207.67@KRBTEST.COM (short username kudu)
I20260929 18:34:54.503501 9636 webserver.cc:460] Webserver started at http://127.28.207.67:33947/ using document root <none> and password file <none>
I20260929 18:34:54.504266 9636 fs_manager.cc:359] Metadata directory not provided
I20260929 18:34:54.504369 9636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:34:54.504631 9636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 18:34:54 dist-test-slave-qcpv krb5kdc[8805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706894, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for HTTP/127.28.207.84@KRBTEST.COM
I20260929 18:34:54.528555 9636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-2/data/instance:
uuid: "f41e08f7f1ca4d69bf315bd3a279688e"
format_stamp: "Formatted at 2026-09-29 18:34:54 on dist-test-slave-qcpv"
server_key: "cd1bfcb50ee02196fb867b09d477fd0d"
server_key_iv: "f501904010807d21dd2bd07fa251e011"
server_key_version: "kuduclusterkey@0"
I20260929 18:34:54.532603 9636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance:
uuid: "f41e08f7f1ca4d69bf315bd3a279688e"
format_stamp: "Formatted at 2026-09-29 18:34:54 on dist-test-slave-qcpv"
server_key: "cd1bfcb50ee02196fb867b09d477fd0d"
server_key_iv: "f501904010807d21dd2bd07fa251e011"
server_key_version: "kuduclusterkey@0"
I20260929 18:34:54.539708 9636 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260929 18:34:54.563870 9651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:54.566344 9636 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260929 18:34:54.566702 9636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-2/wal
uuid: "f41e08f7f1ca4d69bf315bd3a279688e"
format_stamp: "Formatted at 2026-09-29 18:34:54 on dist-test-slave-qcpv"
server_key: "cd1bfcb50ee02196fb867b09d477fd0d"
server_key_iv: "f501904010807d21dd2bd07fa251e011"
server_key_version: "kuduclusterkey@0"
I20260929 18:34:54.567066 9636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:34:54.584123 9636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:34:54.588936 9636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:34:54.589203 9636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:34:54.590085 9636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:34:54.591295 9636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:34:54.591382 9636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:54.591452 9636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:34:54.591508 9636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:54.605737 9636 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.67:33167
I20260929 18:34:54.607378 9764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.67:33167 every 8 connection(s)
I20260929 18:34:54.607584 9636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
I20260929 18:34:54.613920 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 9636
I20260929 18:34:54.614081 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance
I20260929 18:34:54.614450 29501 external_mini_cluster.cc:1428] Setting key e731d69f24ca0bbcd1ac5123fe5dd727
Sep 29 18:34:54 dist-test-slave-qcpv krb5kdc[8805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706894, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:34:54.629719 9499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:54.613729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:36441 (local address 127.28.207.126:34973)
0929 18:34:54.613922 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:54.613927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:54.616671 (+ 2744us) server_negotiation.cc:407] Connection header received
0929 18:34:54.618000 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:54.618004 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:54.618066 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:54.618141 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:34:54.619513 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:54.620108 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:54.621326 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:54.621688 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:54.621809 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:54.625250 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:34:54.625253 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:34:54.625256 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:34:54.625283 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:34:54.626828 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:54.627549 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:54.627553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:54.627554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:54.627642 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:54.628087 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:54.628091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:54.628093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:54.628472 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:34:54.628559 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:54.629377 (+ 818us) server_negotiation.cc:299] Negotiation successful
0929 18:34:54.629521 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260929 18:34:54.631803 9767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:54.614352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:34973 (local address 127.28.207.67:36441)
0929 18:34:54.616488 (+ 2136us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:54.616568 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:54.617767 (+ 1199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:54.618344 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:54.618352 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:54.618681 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:34:54.619366 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:54.619381 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:54.620243 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:54.620252 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:54.621196 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:54.621205 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:54.621860 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:54.621864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:54.622057 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:54.623031 (+ 974us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:34:54.623054 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:34:54.625034 (+ 1980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:34:54.627026 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:34:54.627032 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:34:54.627049 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:34:54.627351 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:34:54.627746 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:34:54.627749 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:34:54.627751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:34:54.627954 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:34:54.628627 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:34:54.628635 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:34:54.629056 (+ 421us) client_negotiation.cc:769] Sending connection context
0929 18:34:54.630039 (+ 983us) client_negotiation.cc:240] Negotiation successful
0929 18:34:54.630485 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1463,"thread_start_us":199,"threads_started":1}
I20260929 18:34:54.633267 9765 heartbeater.cc:347] Connected to a master server at 127.28.207.126:34973
I20260929 18:34:54.633549 9765 heartbeater.cc:464] Registering TS with master...
I20260929 18:34:54.634173 9765 heartbeater.cc:511] Master 127.28.207.126:34973 requested a full tablet report, sending...
I20260929 18:34:54.635456 9281 ts_manager.cc:194] Registered new tserver with Master: f41e08f7f1ca4d69bf315bd3a279688e (127.28.207.67:33167)
I20260929 18:34:54.636415 9281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.67@KRBTEST.COM'} at 127.28.207.67:36441
I20260929 18:34:54.641819 29501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:34:54.643019 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 9362
I20260929 18:34:54.651475 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 9501
I20260929 18:34:54.658547 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 9636
I20260929 18:34:54.669844 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 9249
I20260929 18:34:54.680406 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:34973
--webserver_interface=127.28.207.126
--webserver_port=37279
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:37075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:34973
--ranger_config_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.207.84:42421/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.207.84:42421/kms with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:34:54.802017 9768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:34:54.802313 9768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:34:54.802353 9768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:34:54.806493 9768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:34:54.806577 9768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:34:54.806597 9768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:34:54.806612 9768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:34:54.806628 9768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:34:54.812325 9768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:37075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.207.84:42421/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:34973
--ranger_config_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:34973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=37279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:34:54.813401 9768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:34:54.814419 9768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:34:54.823917 9774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:54.823931 9773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:54.824397 9776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:34:54.825302 9768 server_base.cc:1034] running on GCE node
I20260929 18:34:54.826300 9768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:34:54.828419 9768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:34:54.829680 9768 hybrid_clock.cc:648] HybridClock initialized: now 1790706894829590 us; error 115 us; skew 500 ppm
Sep 29 18:34:54 dist-test-slave-qcpv krb5kdc[8805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706894, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:34:54.833266 9768 init.cc:377] Logged in from keytab as kudu/127.28.207.126@KRBTEST.COM (short username kudu)
I20260929 18:34:54.834486 9768 webserver.cc:460] Webserver started at http://127.28.207.126:37279/ using document root <none> and password file <none>
I20260929 18:34:54.835384 9768 fs_manager.cc:359] Metadata directory not provided
I20260929 18:34:54.835515 9768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 18:34:54 dist-test-slave-qcpv krb5kdc[8805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706894, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for HTTP/127.28.207.84@KRBTEST.COM
I20260929 18:34:54.908432 9768 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 18:34:54.911238 9784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:54.912957 9768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 18:34:54.913113 9768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "cf0a84c3350545f6966cc2b9b3c3dac0"
format_stamp: "Formatted at 2026-09-29 18:34:52 on dist-test-slave-qcpv"
server_key: "b7c140968fb70ed2858701de29bc51f0"
server_key_iv: "50890ccfcb91d179841578f4bce39b23"
server_key_version: "kuduclusterkey@0"
I20260929 18:34:54.913695 9768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:34:54.927111 9768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:34:54.931131 9768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:34:54.931370 9768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:34:54.940768 9768 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:34973
I20260929 18:34:54.940779 9836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:34973 every 8 connection(s)
I20260929 18:34:54.942114 9768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:34:54.947052 9837 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:34:54.948791 9837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0: Bootstrap starting.
I20260929 18:34:54.950099 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 9768
I20260929 18:34:54.950618 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.207.84:42421/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.65:42631
--local_ip_for_outbound_sockets=127.28.207.65
--tserver_master_addrs=127.28.207.126:34973
--webserver_port=39253
--webserver_interface=127.28.207.65
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:37075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.207.84:42421/kms with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:34:54.966835 9837 log.cc:824] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0: Log is configured to *not* fsync() on all Append() calls
I20260929 18:34:54.974256 9837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:34:54.974757 9837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0: Bootstrap complete.
I20260929 18:34:54.979457 9837 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf0a84c3350545f6966cc2b9b3c3dac0" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 34973 } }
I20260929 18:34:54.979954 9837 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf0a84c3350545f6966cc2b9b3c3dac0, State: Initialized, Role: FOLLOWER
I20260929 18:34:54.980542 9837 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf0a84c3350545f6966cc2b9b3c3dac0" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 34973 } }
I20260929 18:34:54.980698 9837 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:34:54.980801 9837 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:34:54.980926 9837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0 [term 1 FOLLOWER]: Advancing to term 2
I20260929 18:34:54.983186 9837 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf0a84c3350545f6966cc2b9b3c3dac0" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 34973 } }
I20260929 18:34:54.983551 9837 leader_election.cc:302] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf0a84c3350545f6966cc2b9b3c3dac0; no voters:
I20260929 18:34:54.983916 9837 leader_election.cc:288] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 18:34:54.984215 9841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 18:34:54.984628 9841 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0 [term 2 LEADER]: Becoming Leader. State: Replica: cf0a84c3350545f6966cc2b9b3c3dac0, State: Running, Role: LEADER
I20260929 18:34:54.985040 9841 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf0a84c3350545f6966cc2b9b3c3dac0" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 34973 } }
I20260929 18:34:54.985466 9837 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:34:54.986899 9842 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cf0a84c3350545f6966cc2b9b3c3dac0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf0a84c3350545f6966cc2b9b3c3dac0" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 34973 } } }
I20260929 18:34:54.987061 9842 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0 [sys.catalog]: This master's current role is: LEADER
I20260929 18:34:54.987351 9843 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf0a84c3350545f6966cc2b9b3c3dac0. Latest consensus state: current_term: 2 leader_uuid: "cf0a84c3350545f6966cc2b9b3c3dac0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf0a84c3350545f6966cc2b9b3c3dac0" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 34973 } } }
I20260929 18:34:54.987440 9843 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0 [sys.catalog]: This master's current role is: LEADER
I20260929 18:34:54.987741 9850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:34:54.991871 9850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:34:54.992565 9850 catalog_manager.cc:1259] Loaded cluster ID: b0c97a7a9286480283f08847528e73d1
I20260929 18:34:54.992535 9837 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 18:34:54.992648 9850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:34:54.995364 9850 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:34:54.999064 9850 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0: Loaded TSK: 0
I20260929 18:34:55.000895 9850 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 18:34:55.142841 9839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:34:55.143600 9839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:34:55.143704 9839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:34:55.151768 9839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:34:55.151880 9839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:34:55.151993 9839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.65
I20260929 18:34:55.161804 9839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:37075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.207.84:42421/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.65:42631
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.207.65
--webserver_port=39253
--tserver_master_addrs=127.28.207.126:34973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.65
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:34:55.163910 9839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:34:55.165766 9839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:34:55.181481 9875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:55.181768 9874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:55.181501 9877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:34:55.182974 9839 server_base.cc:1034] running on GCE node
I20260929 18:34:55.183662 9839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:34:55.184574 9839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:34:55.185763 9839 hybrid_clock.cc:648] HybridClock initialized: now 1790706895185729 us; error 49 us; skew 500 ppm
Sep 29 18:34:55 dist-test-slave-qcpv krb5kdc[8805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706895, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:34:55.189540 9839 init.cc:377] Logged in from keytab as kudu/127.28.207.65@KRBTEST.COM (short username kudu)
I20260929 18:34:55.190634 9839 webserver.cc:460] Webserver started at http://127.28.207.65:39253/ using document root <none> and password file <none>
I20260929 18:34:55.191366 9839 fs_manager.cc:359] Metadata directory not provided
I20260929 18:34:55.191485 9839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 18:34:55 dist-test-slave-qcpv krb5kdc[8805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706895, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for HTTP/127.28.207.84@KRBTEST.COM
I20260929 18:34:55.216462 9839 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260929 18:34:55.222281 9885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:55.223963 9839 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.001s
I20260929 18:34:55.224103 9839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-0/wal
uuid: "a337656f4c7340cca6c6a620452e40d1"
format_stamp: "Formatted at 2026-09-29 18:34:53 on dist-test-slave-qcpv"
server_key: "dbec8b250afec688d70319697804200e"
server_key_iv: "c2676566bfa6ef8cb21ae4b1b7dc3f58"
server_key_version: "kuduclusterkey@0"
I20260929 18:34:55.224771 9839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:34:55.250828 9839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:34:55.254633 9839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:34:55.254901 9839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:34:55.255661 9839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:34:55.256839 9839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:34:55.256915 9839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:55.256980 9839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:34:55.257007 9839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:55.268479 9839 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.65:42631
I20260929 18:34:55.269585 9839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
I20260929 18:34:55.270462 9998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.65:42631 every 8 connection(s)
I20260929 18:34:55.271210 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 9839
I20260929 18:34:55.271646 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.207.84:42421/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.66:44127
--local_ip_for_outbound_sockets=127.28.207.66
--tserver_master_addrs=127.28.207.126:34973
--webserver_port=44809
--webserver_interface=127.28.207.66
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:37075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.207.84:42421/kms with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:34:55 dist-test-slave-qcpv krb5kdc[8805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706895, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:34:55.298430 10003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:55.275282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:34973 (local address 127.28.207.65:34393)
0929 18:34:55.276971 (+ 1689us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:55.277054 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:55.278404 (+ 1350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:55.281219 (+ 2815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:55.281229 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:55.281702 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:34:55.282477 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:55.282621 (+ 144us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:55.285102 (+ 2481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:55.285107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:55.286932 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:55.286950 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:55.288755 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:55.288762 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:55.289000 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:55.289718 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:34:55.289741 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:34:55.291612 (+ 1871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:34:55.294477 (+ 2865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:34:55.294484 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:34:55.294500 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:34:55.294896 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:34:55.295295 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:34:55.295298 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:34:55.295300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:34:55.295464 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:34:55.296295 (+ 831us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:34:55.296302 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:34:55.296605 (+ 303us) client_negotiation.cc:769] Sending connection context
0929 18:34:55.296916 (+ 311us) client_negotiation.cc:240] Negotiation successful
0929 18:34:55.297284 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1385,"thread_start_us":221,"threads_started":1}
I20260929 18:34:55.300257 10002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:55.274669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:34393 (local address 127.28.207.126:34973)
0929 18:34:55.279175 (+ 4506us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:55.279197 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:55.279257 (+ 60us) server_negotiation.cc:407] Connection header received
0929 18:34:55.279635 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:55.279651 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:55.280177 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:55.280614 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:34:55.282902 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:55.284875 (+ 1973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:55.287170 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:55.288247 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:55.288449 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:55.291927 (+ 3478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:34:55.291936 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:34:55.291957 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:34:55.292001 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:34:55.294323 (+ 2322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:55.295041 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:55.295047 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:55.295053 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:55.295194 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:55.295568 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:55.295571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:55.295573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:55.296105 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:34:55.296239 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:55.299048 (+ 2809us) server_negotiation.cc:299] Negotiation successful
0929 18:34:55.299347 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4262,"thread_start_us":156,"threads_started":1}
I20260929 18:34:55.304685 9999 heartbeater.cc:347] Connected to a master server at 127.28.207.126:34973
I20260929 18:34:55.305049 9999 heartbeater.cc:464] Registering TS with master...
I20260929 18:34:55.305866 9999 heartbeater.cc:511] Master 127.28.207.126:34973 requested a full tablet report, sending...
W20260929 18:34:55.308799 9999 heartbeater.cc:644] Failed to heartbeat to 127.28.207.126:34973 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 18:34:55.308956 9999 heartbeater.cc:464] Registering TS with master...
I20260929 18:34:55.309207 9999 heartbeater.cc:511] Master 127.28.207.126:34973 requested a full tablet report, sending...
I20260929 18:34:55.310179 9999 heartbeater.cc:464] Registering TS with master...
I20260929 18:34:55.310374 9999 heartbeater.cc:511] Master 127.28.207.126:34973 requested a full tablet report, sending...
W20260929 18:34:55.311178 9999 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:34:55.430523 10001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:34:55.430980 10001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:34:55.431066 10001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:34:55.436735 10001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:34:55.436843 10001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:34:55.436914 10001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.66
I20260929 18:34:55.443559 10001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:37075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.207.84:42421/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.66:44127
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.207.66
--webserver_port=44809
--tserver_master_addrs=127.28.207.126:34973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.66
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:34:55.444949 10001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:34:55.446223 10001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:34:55.459798 10008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:55.459949 10011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:34:55.460098 10001 server_base.cc:1034] running on GCE node
W20260929 18:34:55.459798 10009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:34:55.460897 10001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:34:55.461807 10001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:34:55.463246 10001 hybrid_clock.cc:648] HybridClock initialized: now 1790706895463220 us; error 49 us; skew 500 ppm
Sep 29 18:34:55 dist-test-slave-qcpv krb5kdc[8805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706895, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:34:55.467314 10001 init.cc:377] Logged in from keytab as kudu/127.28.207.66@KRBTEST.COM (short username kudu)
I20260929 18:34:55.468721 10001 webserver.cc:460] Webserver started at http://127.28.207.66:44809/ using document root <none> and password file <none>
I20260929 18:34:55.469591 10001 fs_manager.cc:359] Metadata directory not provided
I20260929 18:34:55.469763 10001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 18:34:55 dist-test-slave-qcpv krb5kdc[8805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706895, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for HTTP/127.28.207.84@KRBTEST.COM
I20260929 18:34:55.500764 10001 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.002s
I20260929 18:34:55.506925 10019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:55.509184 10001 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.003s
I20260929 18:34:55.509373 10001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-1/wal
uuid: "2004d94307344d29bda62f6cd1500167"
format_stamp: "Formatted at 2026-09-29 18:34:54 on dist-test-slave-qcpv"
server_key: "7dce3e9da78286745e4f5b6f90eb04d7"
server_key_iv: "63311e5f90ecf6ee45752f0c926925d0"
server_key_version: "kuduclusterkey@0"
I20260929 18:34:55.510285 10001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:34:55.554880 10001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:34:55.559628 10001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:34:55.559907 10001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:34:55.560788 10001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:34:55.562099 10001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:34:55.562170 10001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:55.562242 10001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:34:55.562274 10001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:55.576853 10001 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.66:44127
I20260929 18:34:55.578492 10001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
I20260929 18:34:55.578691 10133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.66:44127 every 8 connection(s)
I20260929 18:34:55.582360 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 10001
I20260929 18:34:55.583113 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.28.207.84:42421/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.67:33167
--local_ip_for_outbound_sockets=127.28.207.67
--tserver_master_addrs=127.28.207.126:34973
--webserver_port=33947
--webserver_interface=127.28.207.67
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:37075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.28.207.84:42421/kms with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:34:55 dist-test-slave-qcpv krb5kdc[8805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706895, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:34:55.600407 10002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:55.581776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:44329 (local address 127.28.207.126:34973)
0929 18:34:55.581955 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:55.581960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:55.584048 (+ 2088us) server_negotiation.cc:407] Connection header received
0929 18:34:55.585485 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:55.585490 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:55.585559 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:55.585622 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:34:55.587552 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:55.588477 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:55.590516 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:55.590937 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:55.591036 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:55.594702 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:34:55.594707 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:34:55.594710 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:34:55.594745 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:34:55.596914 (+ 2169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:55.597978 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:55.597982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:55.597985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:55.598097 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:55.598681 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:55.598686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:55.598688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:55.599014 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:34:55.599140 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:55.599992 (+ 852us) server_negotiation.cc:299] Negotiation successful
0929 18:34:55.600196 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 18:34:55.603507 10136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:55.582423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:34973 (local address 127.28.207.66:44329)
0929 18:34:55.583853 (+ 1430us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:55.583916 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:55.585208 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:55.586108 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:55.586118 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:55.586529 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:34:55.587339 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:55.587364 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:55.588661 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:55.588665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:55.590350 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:55.590366 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:55.591277 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:55.591281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:55.591500 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:55.592324 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:34:55.592347 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:34:55.594397 (+ 2050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:34:55.597120 (+ 2723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:34:55.597134 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:34:55.597163 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:34:55.597792 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:34:55.598237 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:34:55.598244 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:34:55.598247 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:34:55.598541 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:34:55.599150 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:34:55.599159 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:34:55.599618 (+ 459us) client_negotiation.cc:769] Sending connection context
0929 18:34:55.601062 (+ 1444us) client_negotiation.cc:240] Negotiation successful
0929 18:34:55.601541 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1158,"thread_start_us":178,"threads_started":1}
I20260929 18:34:55.605618 10134 heartbeater.cc:347] Connected to a master server at 127.28.207.126:34973
I20260929 18:34:55.605937 10134 heartbeater.cc:464] Registering TS with master...
I20260929 18:34:55.606745 10134 heartbeater.cc:511] Master 127.28.207.126:34973 requested a full tablet report, sending...
W20260929 18:34:55.609334 10134 heartbeater.cc:644] Failed to heartbeat to 127.28.207.126:34973 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 18:34:55.609513 10134 heartbeater.cc:464] Registering TS with master...
I20260929 18:34:55.610016 10134 heartbeater.cc:511] Master 127.28.207.126:34973 requested a full tablet report, sending...
I20260929 18:34:55.611075 10134 heartbeater.cc:464] Registering TS with master...
I20260929 18:34:55.611279 10134 heartbeater.cc:511] Master 127.28.207.126:34973 requested a full tablet report, sending...
W20260929 18:34:55.612067 10134 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:34:55.763726 10137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:34:55.764258 10137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:34:55.764359 10137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:34:55.772595 10137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:34:55.772703 10137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:34:55.772815 10137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.67
I20260929 18:34:55.782430 10137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:37075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.28.207.84:42421/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.67:33167
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.207.67
--webserver_port=33947
--tserver_master_addrs=127.28.207.126:34973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.67
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:34:55.784955 10137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:34:55.787144 10137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:34:55.806216 10145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:55.806216 10143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:55.806252 10142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:34:55.807963 10137 server_base.cc:1034] running on GCE node
I20260929 18:34:55.808686 10137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:34:55.809787 10137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:34:55.810962 10137 hybrid_clock.cc:648] HybridClock initialized: now 1790706895810909 us; error 67 us; skew 500 ppm
Sep 29 18:34:55 dist-test-slave-qcpv krb5kdc[8805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706895, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:34:55.815752 10137 init.cc:377] Logged in from keytab as kudu/127.28.207.67@KRBTEST.COM (short username kudu)
I20260929 18:34:55.817134 10137 webserver.cc:460] Webserver started at http://127.28.207.67:33947/ using document root <none> and password file <none>
I20260929 18:34:55.817945 10137 fs_manager.cc:359] Metadata directory not provided
I20260929 18:34:55.818055 10137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 18:34:55 dist-test-slave-qcpv krb5kdc[8805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706895, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for HTTP/127.28.207.84@KRBTEST.COM
I20260929 18:34:55.854102 10137 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.004s sys 0.000s
I20260929 18:34:55.857863 10153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:55.860250 10137 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 18:34:55.860464 10137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-2/wal
uuid: "f41e08f7f1ca4d69bf315bd3a279688e"
format_stamp: "Formatted at 2026-09-29 18:34:54 on dist-test-slave-qcpv"
server_key: "cd1bfcb50ee02196fb867b09d477fd0d"
server_key_iv: "f501904010807d21dd2bd07fa251e011"
server_key_version: "kuduclusterkey@0"
I20260929 18:34:55.861420 10137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:34:55.886435 10137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:34:55.891862 10137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:34:55.892167 10137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:34:55.893146 10137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:34:55.894446 10137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:34:55.894593 10137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:55.894707 10137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:34:55.894747 10137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:55.910447 10137 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.67:33167
I20260929 18:34:55.910622 10266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.67:33167 every 8 connection(s)
I20260929 18:34:55.912618 10137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
I20260929 18:34:55.916539 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 10137
Sep 29 18:34:55 dist-test-slave-qcpv krb5kdc[8805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706895, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:34:55.932844 10278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:55.920257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35216 (local address 127.28.207.126:34973)
0929 18:34:55.920709 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:55.920719 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:55.920736 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:34:55.920787 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:55.920790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:55.920845 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:55.920952 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:34:55.922205 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:55.923096 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:55.925169 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:55.925474 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:55.925605 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:55.927831 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:34:55.927835 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:34:55.927838 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:34:55.927864 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:34:55.930203 (+ 2339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:55.931130 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:55.931136 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:55.931138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:55.931252 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:55.931575 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:55.931580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:55.931582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:55.931827 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:34:55.932069 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:55.932440 (+ 371us) server_negotiation.cc:299] Negotiation successful
0929 18:34:55.932610 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":360,"thread_start_us":138,"threads_started":1}
I20260929 18:34:55.934633 10002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:55.915843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:36159 (local address 127.28.207.126:34973)
0929 18:34:55.916017 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:55.916023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:55.918630 (+ 2607us) server_negotiation.cc:407] Connection header received
0929 18:34:55.920097 (+ 1467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:55.920101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:55.920173 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:55.920281 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:34:55.922404 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:55.923525 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:55.925181 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:55.925685 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:55.925826 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:55.929331 (+ 3505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:34:55.929335 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:34:55.929338 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:34:55.929390 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:34:55.931709 (+ 2319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:55.932443 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:55.932448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:55.932450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:55.932530 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:55.932991 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:55.932995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:55.932996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:55.933204 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:34:55.933283 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:55.934303 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0929 18:34:55.934431 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":59}
I20260929 18:34:55.935665 10276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:55.916388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:34973 (local address 127.28.207.67:36159)
0929 18:34:55.918447 (+ 2059us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:55.918511 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:55.919795 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:55.920773 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:55.920786 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:55.921285 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:34:55.922189 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:55.922212 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:55.923742 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:55.923747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:55.925019 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:55.925031 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:55.925902 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:55.925909 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:55.926144 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:55.926826 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:34:55.926849 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:34:55.929090 (+ 2241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:34:55.931887 (+ 2797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:34:55.931897 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:34:55.931918 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:34:55.932284 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:34:55.932656 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:34:55.932660 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:34:55.932662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:34:55.932861 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:34:55.933402 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:34:55.933413 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:34:55.933807 (+ 394us) client_negotiation.cc:769] Sending connection context
0929 18:34:55.934084 (+ 277us) client_negotiation.cc:240] Negotiation successful
0929 18:34:55.934590 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1798,"thread_start_us":176,"threads_started":1}
I20260929 18:34:55.937250 10267 heartbeater.cc:347] Connected to a master server at 127.28.207.126:34973
I20260929 18:34:55.937518 10267 heartbeater.cc:464] Registering TS with master...
I20260929 18:34:55.938016 10267 heartbeater.cc:511] Master 127.28.207.126:34973 requested a full tablet report, sending...
I20260929 18:34:55.939842 9800 ts_manager.cc:194] Registered new tserver with Master: f41e08f7f1ca4d69bf315bd3a279688e (127.28.207.67:33167)
I20260929 18:34:55.941341 9800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.67@KRBTEST.COM'} at 127.28.207.67:36159
I20260929 18:34:56.311560 9999 heartbeater.cc:464] Registering TS with master...
I20260929 18:34:56.312021 9999 heartbeater.cc:511] Master 127.28.207.126:34973 requested a full tablet report, sending...
I20260929 18:34:56.313200 9801 ts_manager.cc:194] Registered new tserver with Master: a337656f4c7340cca6c6a620452e40d1 (127.28.207.65:42631)
I20260929 18:34:56.314178 9801 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.65@KRBTEST.COM'} at 127.28.207.65:34393
Sep 29 18:34:56 dist-test-slave-qcpv krb5kdc[8805](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790706896, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:34:56.612386 10134 heartbeater.cc:464] Registering TS with master...
I20260929 18:34:56.612787 10134 heartbeater.cc:511] Master 127.28.207.126:34973 requested a full tablet report, sending...
I20260929 18:34:56.614233 9801 ts_manager.cc:194] Registered new tserver with Master: 2004d94307344d29bda62f6cd1500167 (127.28.207.66:44127)
I20260929 18:34:56.615065 9801 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.66@KRBTEST.COM'} at 127.28.207.66:44329
I20260929 18:34:56.618664 29501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:34:56.635121 10288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:56.623311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35218 (local address 127.28.207.126:34973)
0929 18:34:56.623691 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:56.623697 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:56.624044 (+ 347us) server_negotiation.cc:407] Connection header received
0929 18:34:56.624276 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:56.624280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:56.624519 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:56.624631 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:34:56.626070 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:56.626971 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:56.628247 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:56.628611 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:56.628701 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:56.630804 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:34:56.630810 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:34:56.630815 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:34:56.630847 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:34:56.633022 (+ 2175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:56.633566 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:56.633569 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:56.633571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:56.633663 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:56.633999 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:56.634003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:56.634004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:56.634244 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:34:56.634349 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:56.634749 (+ 400us) server_negotiation.cc:299] Negotiation successful
0929 18:34:56.634928 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":155,"threads_started":1}
I20260929 18:34:56.640569 9801 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:34:56.646469 9801 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:34:56.679414 10295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:56.666093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:44127 (local address 127.0.0.1:44762)
0929 18:34:56.670946 (+ 4853us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:56.670960 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:56.671063 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:56.672962 (+ 1899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:56.672969 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:56.672994 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:34:56.673175 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:56.673181 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:56.674938 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:56.674944 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:56.676590 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:56.676614 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:56.678696 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:56.678700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:56.678842 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:56.678934 (+ 92us) client_negotiation.cc:769] Sending connection context
0929 18:34:56.679068 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 18:34:56.679232 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4788,"thread_start_us":148,"threads_started":1}
I20260929 18:34:56.680874 10297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:56.667125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44762 (local address 127.28.207.66:44127)
0929 18:34:56.670781 (+ 3656us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:56.670787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:56.671162 (+ 375us) server_negotiation.cc:407] Connection header received
0929 18:34:56.671266 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:56.671274 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:56.671541 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:56.672723 (+ 1182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:34:56.673311 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:56.674656 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:56.677261 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:56.678281 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:56.678420 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:56.678610 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:56.680576 (+ 1966us) server_negotiation.cc:299] Negotiation successful
0929 18:34:56.680713 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3541,"thread_start_us":116,"threads_started":1}
I20260929 18:34:56.682817 10294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:56.668349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:42631 (local address 127.0.0.1:43396)
0929 18:34:56.674936 (+ 6587us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:56.674952 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:56.676055 (+ 1103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:56.676986 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:56.676989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:56.677007 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:34:56.677123 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:56.677130 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:56.678618 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:56.678621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:56.680873 (+ 2252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:56.680883 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:56.682321 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:56.682325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:56.682428 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:56.682456 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 18:34:56.682545 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 18:34:56.682641 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6494,"thread_start_us":133,"threads_started":1}
I20260929 18:34:56.683768 10299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:56.669083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43396 (local address 127.28.207.65:42631)
0929 18:34:56.670556 (+ 1473us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:56.670565 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:56.676177 (+ 5612us) server_negotiation.cc:407] Connection header received
0929 18:34:56.676286 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:56.676293 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:56.676605 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:56.676780 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:34:56.677259 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:56.678460 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:56.681014 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:56.681948 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:56.682064 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:56.682252 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:56.683469 (+ 1217us) server_negotiation.cc:299] Negotiation successful
0929 18:34:56.683615 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1352,"thread_start_us":139,"threads_started":1}
I20260929 18:34:56.685810 10055 tablet_service.cc:1467] Processing CreateTablet for tablet bb6d82bbf8db4f34b146b6b1c62299f3 (DEFAULT_TABLE table=test-table [id=31fae3a4c494465688cf8104b6d875ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:34:56.688414 10055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb6d82bbf8db4f34b146b6b1c62299f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:34:56.689428 10298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:56.666039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:33167 (local address 127.0.0.1:39070)
0929 18:34:56.669995 (+ 3956us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:56.670070 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:56.670239 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:56.679749 (+ 9510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:56.679755 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:56.679776 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:34:56.679877 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:56.679883 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:56.685748 (+ 5865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:56.685754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:56.687233 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:56.687249 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:56.688918 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:56.688922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:56.689033 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:56.689061 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 18:34:56.689151 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 18:34:56.689246 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3713,"thread_start_us":133,"threads_started":1}
I20260929 18:34:56.689112 9923 tablet_service.cc:1467] Processing CreateTablet for tablet bb6d82bbf8db4f34b146b6b1c62299f3 (DEFAULT_TABLE table=test-table [id=31fae3a4c494465688cf8104b6d875ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:34:56.691592 9923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb6d82bbf8db4f34b146b6b1c62299f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:34:56.695806 10296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:56.667129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39070 (local address 127.28.207.67:33167)
0929 18:34:56.678982 (+ 11853us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:56.679000 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:56.679033 (+ 33us) server_negotiation.cc:407] Connection header received
0929 18:34:56.679128 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:56.679134 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:56.679391 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:56.679575 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:34:56.680134 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:56.681250 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:56.687509 (+ 6259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:56.688519 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:56.688657 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:56.688842 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:56.695448 (+ 6606us) server_negotiation.cc:299] Negotiation successful
0929 18:34:56.695617 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11650,"thread_start_us":91,"threads_started":1}
I20260929 18:34:56.699334 10300 tablet_bootstrap.cc:492] T bb6d82bbf8db4f34b146b6b1c62299f3 P 2004d94307344d29bda62f6cd1500167: Bootstrap starting.
I20260929 18:34:56.699431 10201 tablet_service.cc:1467] Processing CreateTablet for tablet bb6d82bbf8db4f34b146b6b1c62299f3 (DEFAULT_TABLE table=test-table [id=31fae3a4c494465688cf8104b6d875ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:34:56.700791 10201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb6d82bbf8db4f34b146b6b1c62299f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:34:56.703655 10301 tablet_bootstrap.cc:492] T bb6d82bbf8db4f34b146b6b1c62299f3 P a337656f4c7340cca6c6a620452e40d1: Bootstrap starting.
I20260929 18:34:56.704267 10300 tablet_bootstrap.cc:654] T bb6d82bbf8db4f34b146b6b1c62299f3 P 2004d94307344d29bda62f6cd1500167: Neither blocks nor log segments found. Creating new log.
I20260929 18:34:56.705385 10300 log.cc:824] T bb6d82bbf8db4f34b146b6b1c62299f3 P 2004d94307344d29bda62f6cd1500167: Log is configured to *not* fsync() on all Append() calls
I20260929 18:34:56.707432 10301 tablet_bootstrap.cc:654] T bb6d82bbf8db4f34b146b6b1c62299f3 P a337656f4c7340cca6c6a620452e40d1: Neither blocks nor log segments found. Creating new log.
I20260929 18:34:56.708262 10300 tablet_bootstrap.cc:492] T bb6d82bbf8db4f34b146b6b1c62299f3 P 2004d94307344d29bda62f6cd1500167: No bootstrap required, opened a new log
I20260929 18:34:56.708604 10300 ts_tablet_manager.cc:1397] T bb6d82bbf8db4f34b146b6b1c62299f3 P 2004d94307344d29bda62f6cd1500167: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 18:34:56.708909 10301 log.cc:824] T bb6d82bbf8db4f34b146b6b1c62299f3 P a337656f4c7340cca6c6a620452e40d1: Log is configured to *not* fsync() on all Append() calls
I20260929 18:34:56.712381 10301 tablet_bootstrap.cc:492] T bb6d82bbf8db4f34b146b6b1c62299f3 P a337656f4c7340cca6c6a620452e40d1: No bootstrap required, opened a new log
I20260929 18:34:56.712776 10301 ts_tablet_manager.cc:1397] T bb6d82bbf8db4f34b146b6b1c62299f3 P a337656f4c7340cca6c6a620452e40d1: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 18:34:56.714524 10300 raft_consensus.cc:348] T bb6d82bbf8db4f34b146b6b1c62299f3 P 2004d94307344d29bda62f6cd1500167 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41e08f7f1ca4d69bf315bd3a279688e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 33167 } } peers { permanent_uuid: "2004d94307344d29bda62f6cd1500167" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 44127 } } peers { permanent_uuid: "a337656f4c7340cca6c6a620452e40d1" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 42631 } }
I20260929 18:34:56.714936 10300 raft_consensus.cc:374] T bb6d82bbf8db4f34b146b6b1c62299f3 P 2004d94307344d29bda62f6cd1500167 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:34:56.714991 10300 raft_consensus.cc:729] T bb6d82bbf8db4f34b146b6b1c62299f3 P 2004d94307344d29bda62f6cd1500167 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2004d94307344d29bda62f6cd1500167, State: Initialized, Role: FOLLOWER
I20260929 18:34:56.715842 10300 consensus_queue.cc:260] T bb6d82bbf8db4f34b146b6b1c62299f3 P 2004d94307344d29bda62f6cd1500167 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41e08f7f1ca4d69bf315bd3a279688e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 33167 } } peers { permanent_uuid: "2004d94307344d29bda62f6cd1500167" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 44127 } } peers { permanent_uuid: "a337656f4c7340cca6c6a620452e40d1" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 42631 } }
I20260929 18:34:56.717775 10300 ts_tablet_manager.cc:1428] T bb6d82bbf8db4f34b146b6b1c62299f3 P 2004d94307344d29bda62f6cd1500167: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260929 18:34:56.718540 10134 heartbeater.cc:503] Master 127.28.207.126:34973 was elected leader, sending a full tablet report...
I20260929 18:34:56.719560 10301 raft_consensus.cc:348] T bb6d82bbf8db4f34b146b6b1c62299f3 P a337656f4c7340cca6c6a620452e40d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41e08f7f1ca4d69bf315bd3a279688e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 33167 } } peers { permanent_uuid: "2004d94307344d29bda62f6cd1500167" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 44127 } } peers { permanent_uuid: "a337656f4c7340cca6c6a620452e40d1" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 42631 } }
I20260929 18:34:56.719954 10301 raft_consensus.cc:374] T bb6d82bbf8db4f34b146b6b1c62299f3 P a337656f4c7340cca6c6a620452e40d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:34:56.720026 10301 raft_consensus.cc:729] T bb6d82bbf8db4f34b146b6b1c62299f3 P a337656f4c7340cca6c6a620452e40d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a337656f4c7340cca6c6a620452e40d1, State: Initialized, Role: FOLLOWER
I20260929 18:34:56.720902 10304 tablet_bootstrap.cc:492] T bb6d82bbf8db4f34b146b6b1c62299f3 P f41e08f7f1ca4d69bf315bd3a279688e: Bootstrap starting.
I20260929 18:34:56.724326 10304 tablet_bootstrap.cc:654] T bb6d82bbf8db4f34b146b6b1c62299f3 P f41e08f7f1ca4d69bf315bd3a279688e: Neither blocks nor log segments found. Creating new log.
I20260929 18:34:56.725126 10301 consensus_queue.cc:260] T bb6d82bbf8db4f34b146b6b1c62299f3 P a337656f4c7340cca6c6a620452e40d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41e08f7f1ca4d69bf315bd3a279688e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 33167 } } peers { permanent_uuid: "2004d94307344d29bda62f6cd1500167" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 44127 } } peers { permanent_uuid: "a337656f4c7340cca6c6a620452e40d1" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 42631 } }
I20260929 18:34:56.725485 10304 log.cc:824] T bb6d82bbf8db4f34b146b6b1c62299f3 P f41e08f7f1ca4d69bf315bd3a279688e: Log is configured to *not* fsync() on all Append() calls
I20260929 18:34:56.726701 10301 ts_tablet_manager.cc:1428] T bb6d82bbf8db4f34b146b6b1c62299f3 P a337656f4c7340cca6c6a620452e40d1: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20260929 18:34:56.727198 9999 heartbeater.cc:503] Master 127.28.207.126:34973 was elected leader, sending a full tablet report...
I20260929 18:34:56.728681 10304 tablet_bootstrap.cc:492] T bb6d82bbf8db4f34b146b6b1c62299f3 P f41e08f7f1ca4d69bf315bd3a279688e: No bootstrap required, opened a new log
I20260929 18:34:56.728933 10304 ts_tablet_manager.cc:1397] T bb6d82bbf8db4f34b146b6b1c62299f3 P f41e08f7f1ca4d69bf315bd3a279688e: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 18:34:56.734545 10304 raft_consensus.cc:348] T bb6d82bbf8db4f34b146b6b1c62299f3 P f41e08f7f1ca4d69bf315bd3a279688e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41e08f7f1ca4d69bf315bd3a279688e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 33167 } } peers { permanent_uuid: "2004d94307344d29bda62f6cd1500167" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 44127 } } peers { permanent_uuid: "a337656f4c7340cca6c6a620452e40d1" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 42631 } }
I20260929 18:34:56.735117 10304 raft_consensus.cc:374] T bb6d82bbf8db4f34b146b6b1c62299f3 P f41e08f7f1ca4d69bf315bd3a279688e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:34:56.735240 10304 raft_consensus.cc:729] T bb6d82bbf8db4f34b146b6b1c62299f3 P f41e08f7f1ca4d69bf315bd3a279688e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f41e08f7f1ca4d69bf315bd3a279688e, State: Initialized, Role: FOLLOWER
I20260929 18:34:56.735896 10304 consensus_queue.cc:260] T bb6d82bbf8db4f34b146b6b1c62299f3 P f41e08f7f1ca4d69bf315bd3a279688e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41e08f7f1ca4d69bf315bd3a279688e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 33167 } } peers { permanent_uuid: "2004d94307344d29bda62f6cd1500167" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 44127 } } peers { permanent_uuid: "a337656f4c7340cca6c6a620452e40d1" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 42631 } }
I20260929 18:34:56.737202 10267 heartbeater.cc:503] Master 127.28.207.126:34973 was elected leader, sending a full tablet report...
I20260929 18:34:56.737362 10304 ts_tablet_manager.cc:1428] T bb6d82bbf8db4f34b146b6b1c62299f3 P f41e08f7f1ca4d69bf315bd3a279688e: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
W20260929 18:34:56.772466 10000 tablet.cc:2369] T bb6d82bbf8db4f34b146b6b1c62299f3 P a337656f4c7340cca6c6a620452e40d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:34:56.807690 10308 raft_consensus.cc:482] T bb6d82bbf8db4f34b146b6b1c62299f3 P f41e08f7f1ca4d69bf315bd3a279688e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:34:56.808029 10308 raft_consensus.cc:504] T bb6d82bbf8db4f34b146b6b1c62299f3 P f41e08f7f1ca4d69bf315bd3a279688e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41e08f7f1ca4d69bf315bd3a279688e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 33167 } } peers { permanent_uuid: "2004d94307344d29bda62f6cd1500167" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 44127 } } peers { permanent_uuid: "a337656f4c7340cca6c6a620452e40d1" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 42631 } }
I20260929 18:34:56.811584 10308 leader_election.cc:288] T bb6d82bbf8db4f34b146b6b1c62299f3 P f41e08f7f1ca4d69bf315bd3a279688e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2004d94307344d29bda62f6cd1500167 (127.28.207.66:44127), a337656f4c7340cca6c6a620452e40d1 (127.28.207.65:42631)
I20260929 18:34:56.817912 10299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:56.811693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:44047 (local address 127.28.207.65:42631)
0929 18:34:56.811873 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:56.811879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:56.812456 (+ 577us) server_negotiation.cc:407] Connection header received
0929 18:34:56.812683 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:56.812686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:56.812742 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:56.812790 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:34:56.813655 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:56.814539 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:56.816354 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:56.817020 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:56.817142 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:56.817201 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:56.817680 (+ 479us) server_negotiation.cc:299] Negotiation successful
0929 18:34:56.817763 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 18:34:56.818259 10297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:56.811532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:51485 (local address 127.28.207.66:44127)
0929 18:34:56.811857 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:56.811861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:56.812128 (+ 267us) server_negotiation.cc:407] Connection header received
0929 18:34:56.812622 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:56.812627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:56.812717 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:56.812787 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:34:56.813410 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:56.814441 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:56.816415 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:56.817119 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:56.817245 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:56.817322 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:56.817972 (+ 650us) server_negotiation.cc:299] Negotiation successful
0929 18:34:56.818073 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232}
I20260929 18:34:56.818500 10309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:56.811094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:42631 (local address 127.28.207.67:44047)
0929 18:34:56.812343 (+ 1249us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:56.812361 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:56.812551 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:56.812939 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:56.812942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:56.812957 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:34:56.813280 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:56.813289 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:56.814696 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:56.814699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:56.816196 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:56.816207 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:56.817425 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:56.817429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:56.817538 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:56.817564 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 18:34:56.818148 (+ 584us) client_negotiation.cc:240] Negotiation successful
0929 18:34:56.818253 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1163,"mutex_wait_us":95,"thread_start_us":180,"threads_started":1}
I20260929 18:34:56.819730 10310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:56.810991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:44127 (local address 127.28.207.67:51485)
0929 18:34:56.812001 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:56.812016 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:56.812172 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:56.812976 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:56.812979 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:56.813017 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:34:56.813261 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:56.813268 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:56.814890 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:56.814894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:56.816244 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:56.816257 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:56.817569 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:56.817587 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:56.817795 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:56.817825 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 18:34:56.819431 (+ 1606us) client_negotiation.cc:240] Negotiation successful
0929 18:34:56.819539 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":932,"mutex_wait_us":10,"thread_start_us":146,"threads_started":1}
I20260929 18:34:56.819754 9946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb6d82bbf8db4f34b146b6b1c62299f3" candidate_uuid: "f41e08f7f1ca4d69bf315bd3a279688e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a337656f4c7340cca6c6a620452e40d1" is_pre_election: true
I20260929 18:34:56.820113 9946 raft_consensus.cc:2454] T bb6d82bbf8db4f34b146b6b1c62299f3 P a337656f4c7340cca6c6a620452e40d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f41e08f7f1ca4d69bf315bd3a279688e in term 0.
I20260929 18:34:56.820693 10154 leader_election.cc:302] T bb6d82bbf8db4f34b146b6b1c62299f3 P f41e08f7f1ca4d69bf315bd3a279688e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a337656f4c7340cca6c6a620452e40d1, f41e08f7f1ca4d69bf315bd3a279688e; no voters:
I20260929 18:34:56.821062 10308 raft_consensus.cc:2781] T bb6d82bbf8db4f34b146b6b1c62299f3 P f41e08f7f1ca4d69bf315bd3a279688e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:34:56.821101 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb6d82bbf8db4f34b146b6b1c62299f3" candidate_uuid: "f41e08f7f1ca4d69bf315bd3a279688e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2004d94307344d29bda62f6cd1500167" is_pre_election: true
I20260929 18:34:56.821213 10308 raft_consensus.cc:482] T bb6d82bbf8db4f34b146b6b1c62299f3 P f41e08f7f1ca4d69bf315bd3a279688e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:34:56.821266 10308 raft_consensus.cc:3037] T bb6d82bbf8db4f34b146b6b1c62299f3 P f41e08f7f1ca4d69bf315bd3a279688e [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:34:56.821429 10088 raft_consensus.cc:2454] T bb6d82bbf8db4f34b146b6b1c62299f3 P 2004d94307344d29bda62f6cd1500167 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f41e08f7f1ca4d69bf315bd3a279688e in term 0.
I20260929 18:34:56.823307 10308 raft_consensus.cc:504] T bb6d82bbf8db4f34b146b6b1c62299f3 P f41e08f7f1ca4d69bf315bd3a279688e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41e08f7f1ca4d69bf315bd3a279688e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 33167 } } peers { permanent_uuid: "2004d94307344d29bda62f6cd1500167" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 44127 } } peers { permanent_uuid: "a337656f4c7340cca6c6a620452e40d1" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 42631 } }
I20260929 18:34:56.824645 10088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb6d82bbf8db4f34b146b6b1c62299f3" candidate_uuid: "f41e08f7f1ca4d69bf315bd3a279688e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2004d94307344d29bda62f6cd1500167"
I20260929 18:34:56.824813 10088 raft_consensus.cc:3037] T bb6d82bbf8db4f34b146b6b1c62299f3 P 2004d94307344d29bda62f6cd1500167 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:34:56.825034 10308 leader_election.cc:288] T bb6d82bbf8db4f34b146b6b1c62299f3 P f41e08f7f1ca4d69bf315bd3a279688e [CANDIDATE]: Term 1 election: Requested vote from peers 2004d94307344d29bda62f6cd1500167 (127.28.207.66:44127), a337656f4c7340cca6c6a620452e40d1 (127.28.207.65:42631)
I20260929 18:34:56.825666 9946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb6d82bbf8db4f34b146b6b1c62299f3" candidate_uuid: "f41e08f7f1ca4d69bf315bd3a279688e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a337656f4c7340cca6c6a620452e40d1"
I20260929 18:34:56.825852 9946 raft_consensus.cc:3037] T bb6d82bbf8db4f34b146b6b1c62299f3 P a337656f4c7340cca6c6a620452e40d1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:34:56.826391 10088 raft_consensus.cc:2454] T bb6d82bbf8db4f34b146b6b1c62299f3 P 2004d94307344d29bda62f6cd1500167 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f41e08f7f1ca4d69bf315bd3a279688e in term 1.
I20260929 18:34:56.827934 9946 raft_consensus.cc:2454] T bb6d82bbf8db4f34b146b6b1c62299f3 P a337656f4c7340cca6c6a620452e40d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f41e08f7f1ca4d69bf315bd3a279688e in term 1.
I20260929 18:34:56.828580 10156 leader_election.cc:302] T bb6d82bbf8db4f34b146b6b1c62299f3 P f41e08f7f1ca4d69bf315bd3a279688e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2004d94307344d29bda62f6cd1500167, f41e08f7f1ca4d69bf315bd3a279688e; no voters:
I20260929 18:34:56.829192 10308 raft_consensus.cc:2781] T bb6d82bbf8db4f34b146b6b1c62299f3 P f41e08f7f1ca4d69bf315bd3a279688e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:34:56.829515 10308 raft_consensus.cc:686] T bb6d82bbf8db4f34b146b6b1c62299f3 P f41e08f7f1ca4d69bf315bd3a279688e [term 1 LEADER]: Becoming Leader. State: Replica: f41e08f7f1ca4d69bf315bd3a279688e, State: Running, Role: LEADER
Sep 29 18:34:56 dist-test-slave-qcpv krb5kdc[8805](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706896, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for HTTP/127.28.207.84@KRBTEST.COM
I20260929 18:34:56.830740 10308 consensus_queue.cc:237] T bb6d82bbf8db4f34b146b6b1c62299f3 P f41e08f7f1ca4d69bf315bd3a279688e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41e08f7f1ca4d69bf315bd3a279688e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 33167 } } peers { permanent_uuid: "2004d94307344d29bda62f6cd1500167" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 44127 } } peers { permanent_uuid: "a337656f4c7340cca6c6a620452e40d1" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 42631 } }
W20260929 18:34:56.830878 10135 tablet.cc:2369] T bb6d82bbf8db4f34b146b6b1c62299f3 P 2004d94307344d29bda62f6cd1500167: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:34:56.835208 9801 catalog_manager.cc:5495] T bb6d82bbf8db4f34b146b6b1c62299f3 P f41e08f7f1ca4d69bf315bd3a279688e reported cstate change: term changed from 0 to 1, leader changed from <none> to f41e08f7f1ca4d69bf315bd3a279688e (127.28.207.67). New cstate: current_term: 1 leader_uuid: "f41e08f7f1ca4d69bf315bd3a279688e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41e08f7f1ca4d69bf315bd3a279688e" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 33167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2004d94307344d29bda62f6cd1500167" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 44127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a337656f4c7340cca6c6a620452e40d1" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 42631 } health_report { overall_health: UNKNOWN } } }
I20260929 18:34:56.872422 10296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:56.866124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39076 (local address 127.28.207.67:33167)
0929 18:34:56.866280 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:56.866286 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:56.866331 (+ 45us) server_negotiation.cc:407] Connection header received
0929 18:34:56.866593 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:56.866597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:56.866669 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:56.866802 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:34:56.867206 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:56.868021 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:56.869411 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:56.869888 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:56.870028 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:56.870513 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:34:56.871303 (+ 790us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:34:56.871440 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:56.872155 (+ 715us) server_negotiation.cc:299] Negotiation successful
0929 18:34:56.872232 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 18:34:56.881253 9946 raft_consensus.cc:1261] T bb6d82bbf8db4f34b146b6b1c62299f3 P a337656f4c7340cca6c6a620452e40d1 [term 1 FOLLOWER]: Refusing update from remote peer f41e08f7f1ca4d69bf315bd3a279688e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:34:56.881253 10088 raft_consensus.cc:1261] T bb6d82bbf8db4f34b146b6b1c62299f3 P 2004d94307344d29bda62f6cd1500167 [term 1 FOLLOWER]: Refusing update from remote peer f41e08f7f1ca4d69bf315bd3a279688e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:34:56.882164 10308 consensus_queue.cc:1035] T bb6d82bbf8db4f34b146b6b1c62299f3 P f41e08f7f1ca4d69bf315bd3a279688e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2004d94307344d29bda62f6cd1500167" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 44127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:34:56.882788 10308 consensus_queue.cc:1035] T bb6d82bbf8db4f34b146b6b1c62299f3 P f41e08f7f1ca4d69bf315bd3a279688e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a337656f4c7340cca6c6a620452e40d1" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 42631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:34:56.896646 10314 mvcc.cc:205] Tried to move back new op lower bound from 7334735449615597568 to 7334735449423069184. Current Snapshot: MvccSnapshot[applied={T|T < 7334735449615597568}]
I20260929 18:34:56.900550 10315 mvcc.cc:205] Tried to move back new op lower bound from 7334735449615597568 to 7334735449423069184. Current Snapshot: MvccSnapshot[applied={T|T < 7334735449615597568}]
I20260929 18:34:56.905441 10316 mvcc.cc:205] Tried to move back new op lower bound from 7334735449615597568 to 7334735449423069184. Current Snapshot: MvccSnapshot[applied={T|T < 7334735449615597568}]
I20260929 18:34:56.915877 9801 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:34:56.916208 9801 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:34:56.919761 9801 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf0a84c3350545f6966cc2b9b3c3dac0: Sending DeleteTablet for 3 replicas of tablet bb6d82bbf8db4f34b146b6b1c62299f3
I20260929 18:34:56.921664 10201 tablet_service.cc:1514] Processing DeleteTablet for tablet bb6d82bbf8db4f34b146b6b1c62299f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:34:56 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:39070
I20260929 18:34:56.922222 9923 tablet_service.cc:1514] Processing DeleteTablet for tablet bb6d82bbf8db4f34b146b6b1c62299f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:34:56 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:43396
I20260929 18:34:56.923066 10055 tablet_service.cc:1514] Processing DeleteTablet for tablet bb6d82bbf8db4f34b146b6b1c62299f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:34:56 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:44762
I20260929 18:34:56.923839 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 9839
I20260929 18:34:56.931227 10325 tablet_replica.cc:330] T bb6d82bbf8db4f34b146b6b1c62299f3 P 2004d94307344d29bda62f6cd1500167: stopping tablet replica
I20260929 18:34:56.931819 10325 raft_consensus.cc:2229] T bb6d82bbf8db4f34b146b6b1c62299f3 P 2004d94307344d29bda62f6cd1500167 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:34:56.932389 10325 raft_consensus.cc:2258] T bb6d82bbf8db4f34b146b6b1c62299f3 P 2004d94307344d29bda62f6cd1500167 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:34:56.935695 10323 tablet_replica.cc:330] T bb6d82bbf8db4f34b146b6b1c62299f3 P f41e08f7f1ca4d69bf315bd3a279688e: stopping tablet replica
I20260929 18:34:56.936167 10323 raft_consensus.cc:2229] T bb6d82bbf8db4f34b146b6b1c62299f3 P f41e08f7f1ca4d69bf315bd3a279688e [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:34:56.937186 10323 raft_consensus.cc:2258] T bb6d82bbf8db4f34b146b6b1c62299f3 P f41e08f7f1ca4d69bf315bd3a279688e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:34:56.939260 10323 ts_tablet_manager.cc:1905] T bb6d82bbf8db4f34b146b6b1c62299f3 P f41e08f7f1ca4d69bf315bd3a279688e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:34:56.945238 10325 ts_tablet_manager.cc:1905] T bb6d82bbf8db4f34b146b6b1c62299f3 P 2004d94307344d29bda62f6cd1500167: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:34:56.953441 10325 ts_tablet_manager.cc:1918] T bb6d82bbf8db4f34b146b6b1c62299f3 P 2004d94307344d29bda62f6cd1500167: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:34:56.953611 10325 log.cc:1196] T bb6d82bbf8db4f34b146b6b1c62299f3 P 2004d94307344d29bda62f6cd1500167: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-1/wal/wals/bb6d82bbf8db4f34b146b6b1c62299f3
I20260929 18:34:56.954015 10325 ts_tablet_manager.cc:1939] T bb6d82bbf8db4f34b146b6b1c62299f3 P 2004d94307344d29bda62f6cd1500167: Deleting consensus metadata
W20260929 18:34:56.955358 9785 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.207.65:42631 (error 108)
I20260929 18:34:56.942793 10323 ts_tablet_manager.cc:1918] T bb6d82bbf8db4f34b146b6b1c62299f3 P f41e08f7f1ca4d69bf315bd3a279688e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:34:56.956382 10323 log.cc:1196] T bb6d82bbf8db4f34b146b6b1c62299f3 P f41e08f7f1ca4d69bf315bd3a279688e: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790706718260899-29501-0/minicluster-data/ts-2/wal/wals/bb6d82bbf8db4f34b146b6b1c62299f3
I20260929 18:34:56.956808 10323 ts_tablet_manager.cc:1939] T bb6d82bbf8db4f34b146b6b1c62299f3 P f41e08f7f1ca4d69bf315bd3a279688e: Deleting consensus metadata
I20260929 18:34:56.959067 9785 catalog_manager.cc:4841] TS f41e08f7f1ca4d69bf315bd3a279688e (127.28.207.67:33167): tablet bb6d82bbf8db4f34b146b6b1c62299f3 (table test-table [id=31fae3a4c494465688cf8104b6d875ad]) successfully deleted
I20260929 18:34:56.959545 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 10001
I20260929 18:34:56.961088 10294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:56.960775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:42631 (local address 127.0.0.1:43412)
0929 18:34:56.960915 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:56.961003 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.65:42631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260929 18:34:56.961404 9785 catalog_manager.cc:4584] TS a337656f4c7340cca6c6a620452e40d1 (127.28.207.65:42631): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bb6d82bbf8db4f34b146b6b1c62299f3: Network error: Client connection negotiation failed: client connection to 127.28.207.65:42631: connect: Connection refused (error 111)
W20260929 18:34:56.970477 10294 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:34:56.969788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:44127 (local address 127.0.0.1:44772)
0929 18:34:56.969978 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:56.970009 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:56.970107 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:56.970328 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.66:44127: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":106}
I20260929 18:34:56.971168 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 10137
I20260929 18:34:56.980916 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 9768
I20260929 18:34:56.990110 29501 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 18:34:57.337538 29501 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 18:34:57.337773 29501 mini_ranger.cc:66] Stopping Ranger...
I20260929 18:34:57.685097 29501 mini_ranger.cc:68] Stopped Ranger
2026-09-29 18:34:57.685 UTC [8835] LOG: received smart shutdown request
2026-09-29 18:34:57.689 UTC [8835] LOG: background worker "logical replication launcher" (PID 8844) exited with exit code 1
2026-09-29 18:34:57.689 UTC [8839] LOG: shutting down
2026-09-29 18:34:57.906 UTC [8835] LOG: database system is shut down
2026-09-29T18:34:57Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (56081 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:34:58 dist-test-slave-qcpv krb5kdc[10335](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:34:58 dist-test-slave-qcpv krb5kdc[10335](info): set up 2 sockets
Sep 29 18:34:58 dist-test-slave-qcpv krb5kdc[10335](info): commencing operation
krb5kdc: starting...
W20260929 18:34:59.183357 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:34:59 dist-test-slave-qcpv krb5kdc[10335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:34:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:34:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.207.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
I20260929 18:34:59.327797 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:35695
--webserver_interface=127.28.207.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:41571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:35695,127.28.207.125:46659,127.28.207.124:32901
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:34:59.446211 10348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:34:59.446540 10348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:34:59.446615 10348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:34:59.450681 10348 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:34:59.450788 10348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:34:59.450825 10348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:34:59.450893 10348 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:34:59.450956 10348 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:34:59.456365 10348 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:41571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:35695,127.28.207.125:46659,127.28.207.124:32901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:35695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:34:59.457515 10348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:34:59.458678 10348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:34:59.468473 10353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:59.468430 10356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:59.468454 10354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:34:59.468868 10348 server_base.cc:1034] running on GCE node
I20260929 18:34:59.469600 10348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:34:59.471267 10348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:34:59.472537 10348 hybrid_clock.cc:648] HybridClock initialized: now 1790706899472504 us; error 54 us; skew 500 ppm
Sep 29 18:34:59 dist-test-slave-qcpv krb5kdc[10335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706899, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:34:59.476576 10348 init.cc:377] Logged in from keytab as kudu/127.28.207.126@KRBTEST.COM (short username kudu)
I20260929 18:34:59.477766 10348 webserver.cc:460] Webserver started at http://127.28.207.126:36801/ using document root <none> and password file <none>
I20260929 18:34:59.478519 10348 fs_manager.cc:359] Metadata directory not provided
I20260929 18:34:59.478636 10348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:34:59.479001 10348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:34:59.481772 10348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "d8d82a3df62548b0b732a58cf70784c8"
format_stamp: "Formatted at 2026-09-29 18:34:59 on dist-test-slave-qcpv"
server_key: "c7e23fade03f71d2007f2eaf0391e374"
server_key_iv: "7b38f3adacb7963f56ccaaf3dd80a988"
server_key_version: "encryptionkey@0"
I20260929 18:34:59.482499 10348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "d8d82a3df62548b0b732a58cf70784c8"
format_stamp: "Formatted at 2026-09-29 18:34:59 on dist-test-slave-qcpv"
server_key: "c7e23fade03f71d2007f2eaf0391e374"
server_key_iv: "7b38f3adacb7963f56ccaaf3dd80a988"
server_key_version: "encryptionkey@0"
I20260929 18:34:59.487159 10348 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.001s
I20260929 18:34:59.490406 10363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:59.492118 10348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:34:59.492401 10348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "d8d82a3df62548b0b732a58cf70784c8"
format_stamp: "Formatted at 2026-09-29 18:34:59 on dist-test-slave-qcpv"
server_key: "c7e23fade03f71d2007f2eaf0391e374"
server_key_iv: "7b38f3adacb7963f56ccaaf3dd80a988"
server_key_version: "encryptionkey@0"
I20260929 18:34:59.492556 10348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:34:59.514600 10348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:34:59.518256 10348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:34:59.518504 10348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:34:59.527748 10348 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:35695
I20260929 18:34:59.528069 10415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:35695 every 8 connection(s)
I20260929 18:34:59.529062 10348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:34:59.532267 10416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:34:59.536655 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 10348
I20260929 18:34:59.536799 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:34:59.536461 10416 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.207.126" port: 35695 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:34:59.537129 29501 external_mini_cluster.cc:1428] Setting key edc81587ca155bf82a55048529bbc95e
WARNING: no policy specified for kudu/127.28.207.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.125@KRBTEST.COM" created.
Sep 29 18:34:59 dist-test-slave-qcpv krb5kdc[10335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706899, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:34:59.557682 10420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:59.539891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57566 (local address 127.28.207.126:35695)
0929 18:34:59.540629 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:59.540651 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:59.540780 (+ 129us) server_negotiation.cc:407] Connection header received
0929 18:34:59.542227 (+ 1447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:59.542237 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:59.542544 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:59.542837 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:34:59.543974 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.545565 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.547129 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.547580 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.547737 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:59.551625 (+ 3888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:34:59.551633 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:34:59.551647 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:34:59.551681 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:34:59.553641 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:59.554485 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:59.554490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:59.554492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:59.554648 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:59.555171 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:59.555182 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:59.555184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:59.555680 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:34:59.555802 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:59.556474 (+ 672us) server_negotiation.cc:299] Negotiation successful
0929 18:34:59.556743 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"spinlock_wait_cycles":48000,"thread_start_us":201,"threads_started":1}
I20260929 18:34:59.557689 10419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:59.539890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:35695 (local address 127.0.0.1:57566)
0929 18:34:59.540559 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:59.540624 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:59.541818 (+ 1194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:59.542981 (+ 1163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:59.542987 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:59.543432 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:34:59.543779 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:59.543793 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.545786 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.545790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:59.546945 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:59.546965 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.547814 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.547818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:59.547941 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:59.548885 (+ 944us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:34:59.548905 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:34:59.551372 (+ 2467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:34:59.553813 (+ 2441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:34:59.553828 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:34:59.553844 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:34:59.554282 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:34:59.554768 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:34:59.554774 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:34:59.554779 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:34:59.555012 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:34:59.555827 (+ 815us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:34:59.555838 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:34:59.556100 (+ 262us) client_negotiation.cc:769] Sending connection context
0929 18:34:59.556344 (+ 244us) client_negotiation.cc:240] Negotiation successful
0929 18:34:59.556982 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":203,"threads_started":1}
I20260929 18:34:59.560653 10416 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.207.125" port: 46659 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:34:59.561764 10419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:59.561304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.125:46659 (local address 127.0.0.1:33788)
0929 18:34:59.561558 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:59.561657 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.125:46659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260929 18:34:59.562039 10367 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.207.125:46659: connect: Connection refused (error 111)
I20260929 18:34:59.563328 10419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:59.563077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.125:46659 (local address 127.0.0.1:33790)
0929 18:34:59.563201 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:59.563265 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.125:46659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 18:34:59.563509 10416 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.207.125:46659: Network error: Client connection negotiation failed: client connection to 127.28.207.125:46659: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.125.keytab.
Entry for principal kudu/127.28.207.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.125.keytab.
I20260929 18:34:59.570469 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.125:46659
--webserver_interface=127.28.207.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.125.keytab
--principal=kudu/127.28.207.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:41571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:35695,127.28.207.125:46659,127.28.207.124:32901
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:34:59.612828 10416 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.207.125" port: 46659 } attempt: 1
I20260929 18:34:59.614200 10419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:59.613610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.125:46659 (local address 127.0.0.1:33798)
0929 18:34:59.613921 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:59.614089 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.125:46659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260929 18:34:59.615249 10419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:59.615022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.125:46659 (local address 127.0.0.1:33808)
0929 18:34:59.615130 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:59.615187 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.125:46659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 18:34:59.615430 10416 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.207.125:46659: Network error: Client connection negotiation failed: client connection to 127.28.207.125:46659: connect: Connection refused (error 111)
I20260929 18:34:59.683630 10416 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.207.125" port: 46659 } attempt: 2
I20260929 18:34:59.684852 10419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:59.684386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.125:46659 (local address 127.0.0.1:33812)
0929 18:34:59.684583 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:59.684660 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.125:46659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 18:34:59.685765 10419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:59.685522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.125:46659 (local address 127.0.0.1:33824)
0929 18:34:59.685647 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:59.685705 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.125:46659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 18:34:59.686022 10416 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.207.125:46659: Network error: Client connection negotiation failed: client connection to 127.28.207.125:46659: connect: Connection refused (error 111)
W20260929 18:34:59.686115 10423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:34:59.686458 10423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:34:59.686544 10423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:34:59.690711 10423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:34:59.690793 10423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:34:59.690817 10423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:34:59.690837 10423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:34:59.690881 10423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:34:59.696440 10423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:41571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:35695,127.28.207.125:46659,127.28.207.124:32901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.125.keytab
--principal=kudu/127.28.207.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.125:46659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.207.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:34:59.697821 10423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:34:59.698992 10423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:34:59.708527 10431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:59.708542 10429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:34:59.708901 10423 server_base.cc:1034] running on GCE node
W20260929 18:34:59.708550 10428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:34:59.709656 10423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:34:59.710934 10423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:34:59.712220 10423 hybrid_clock.cc:648] HybridClock initialized: now 1790706899712155 us; error 87 us; skew 500 ppm
Sep 29 18:34:59 dist-test-slave-qcpv krb5kdc[10335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706899, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:34:59.715943 10423 init.cc:377] Logged in from keytab as kudu/127.28.207.125@KRBTEST.COM (short username kudu)
I20260929 18:34:59.717031 10423 webserver.cc:460] Webserver started at http://127.28.207.125:35687/ using document root <none> and password file <none>
I20260929 18:34:59.717650 10423 fs_manager.cc:359] Metadata directory not provided
I20260929 18:34:59.717727 10423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:34:59.717976 10423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:34:59.720343 10423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-1/data/instance:
uuid: "648eb2f09f534d12b84bb37df0882df7"
format_stamp: "Formatted at 2026-09-29 18:34:59 on dist-test-slave-qcpv"
server_key: "2a7a1dd4da600b0ac23424e179e4208a"
server_key_iv: "991ccbdc1fbe4904b64b09f2398dab25"
server_key_version: "encryptionkey@0"
I20260929 18:34:59.721200 10423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-1/wal/instance:
uuid: "648eb2f09f534d12b84bb37df0882df7"
format_stamp: "Formatted at 2026-09-29 18:34:59 on dist-test-slave-qcpv"
server_key: "2a7a1dd4da600b0ac23424e179e4208a"
server_key_iv: "991ccbdc1fbe4904b64b09f2398dab25"
server_key_version: "encryptionkey@0"
I20260929 18:34:59.727751 10423 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260929 18:34:59.731577 10438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:59.733242 10423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:34:59.733414 10423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-1/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-1/wal
uuid: "648eb2f09f534d12b84bb37df0882df7"
format_stamp: "Formatted at 2026-09-29 18:34:59 on dist-test-slave-qcpv"
server_key: "2a7a1dd4da600b0ac23424e179e4208a"
server_key_iv: "991ccbdc1fbe4904b64b09f2398dab25"
server_key_version: "encryptionkey@0"
I20260929 18:34:59.733592 10423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:34:59.756609 10423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:34:59.760259 10423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:34:59.760529 10423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:34:59.769413 10423 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.125:46659
I20260929 18:34:59.769873 10490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.125:46659 every 8 connection(s)
I20260929 18:34:59.770682 10423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-1/data/info.pb
I20260929 18:34:59.774628 10491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:34:59.777236 10416 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.207.125" port: 46659 } attempt: 3
I20260929 18:34:59.779623 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 10423
I20260929 18:34:59.779784 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-1/wal/instance
I20260929 18:34:59.780110 29501 external_mini_cluster.cc:1428] Setting key 005037fef04a2120e81e0ecb53ce0aa0
I20260929 18:34:59.780313 10491 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.207.126" port: 35695 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:34:59 dist-test-slave-qcpv krb5kdc[10335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706899, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for kudu/127.28.207.125@KRBTEST.COM
Sep 29 18:34:59 dist-test-slave-qcpv krb5kdc[10335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706899, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.125@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.207.124@KRBTEST.COM; defaulting to no policy
I20260929 18:34:59.795060 10419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:59.778062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.125:46659 (local address 127.0.0.1:33826)
0929 18:34:59.778321 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:59.778342 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:59.778459 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:59.782283 (+ 3824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:59.782287 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:59.782520 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:34:59.782667 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:59.782675 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.784437 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.784441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:59.785264 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:59.785277 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.786106 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.786109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:59.786209 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:59.787029 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:34:59.787045 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:34:59.788596 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:34:59.792536 (+ 3940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:34:59.792542 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:34:59.792544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:34:59.792767 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:34:59.793137 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:34:59.793140 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:34:59.793142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:34:59.793271 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:34:59.794164 (+ 893us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:34:59.794167 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:34:59.794293 (+ 126us) client_negotiation.cc:769] Sending connection context
0929 18:34:59.794508 (+ 215us) client_negotiation.cc:240] Negotiation successful
0929 18:34:59.794833 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.124@KRBTEST.COM" created.
I20260929 18:34:59.796711 10493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:59.779104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33826 (local address 127.28.207.125:46659)
0929 18:34:59.779897 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:59.779926 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:59.779994 (+ 68us) server_negotiation.cc:407] Connection header received
0929 18:34:59.781323 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:59.781354 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:59.781934 (+ 580us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:59.782346 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:34:59.782936 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.784273 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.785430 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.785964 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.786144 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:59.788826 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:34:59.788832 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:34:59.788844 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:34:59.788876 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:34:59.792401 (+ 3525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:59.792892 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:59.792898 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:59.792905 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:59.793024 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:59.793391 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:59.793395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:59.793397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:59.794035 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:34:59.794184 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:59.794909 (+ 725us) server_negotiation.cc:299] Negotiation successful
0929 18:34:59.795370 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":205,"threads_started":1}
I20260929 18:34:59.797827 10495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:59.783986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:35695 (local address 127.0.0.1:57568)
0929 18:34:59.784676 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:59.784743 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:59.784940 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:59.785307 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:59.785315 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:59.785930 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:34:59.786186 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:59.786199 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.787543 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.787550 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:59.788794 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:59.788813 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.789478 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.789483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:59.789648 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:59.790252 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:34:59.790276 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:34:59.792147 (+ 1871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:34:59.795592 (+ 3445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:34:59.795604 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:34:59.795613 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:34:59.795918 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:34:59.796239 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:34:59.796244 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:34:59.796246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:34:59.796344 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:34:59.797077 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:34:59.797084 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:34:59.797247 (+ 163us) client_negotiation.cc:769] Sending connection context
0929 18:34:59.797420 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 18:34:59.797665 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":117,"threads_started":1}
I20260929 18:34:59.798777 10420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:59.784102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57568 (local address 127.28.207.126:35695)
0929 18:34:59.784285 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:59.784293 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:59.784877 (+ 584us) server_negotiation.cc:407] Connection header received
0929 18:34:59.785055 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:59.785058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:59.785109 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:59.785198 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:34:59.786424 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.787374 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.788950 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.789345 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.789473 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:59.793152 (+ 3679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:34:59.793158 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:34:59.793162 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:34:59.793227 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:34:59.795405 (+ 2178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:59.796033 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:59.796037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:59.796039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:59.796138 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:59.796509 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:59.796513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:59.796515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:59.796734 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:34:59.797049 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:59.798266 (+ 1217us) server_negotiation.cc:299] Negotiation successful
0929 18:34:59.798502 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 18:34:59.800355 10491 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.207.125" port: 46659 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:34:59.801074 10416 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.207.124" port: 32901 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:34:59.801841 10419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:59.801567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:32901 (local address 127.0.0.1:48598)
0929 18:34:59.801697 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:59.801761 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.124:32901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 18:34:59.802680 10419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:59.802386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:32901 (local address 127.0.0.1:48600)
0929 18:34:59.802565 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:59.802615 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.124:32901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260929 18:34:59.802960 10416 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.207.124:32901: Network error: Client connection negotiation failed: client connection to 127.28.207.124:32901: connect: Connection refused (error 111)
Sep 29 18:34:59 dist-test-slave-qcpv krb5kdc[10335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706899, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.125@KRBTEST.COM for kudu/127.28.207.125@KRBTEST.COM
I20260929 18:34:59.812376 10495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:59.801111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.125:46659 (local address 127.0.0.1:33828)
0929 18:34:59.801269 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:59.801288 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 18:34:59.801387 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:34:59.801843 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:34:59.801849 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:34:59.802079 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:34:59.802228 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:59.802238 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.803421 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.803428 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:59.804640 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:34:59.804658 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.805235 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.805242 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:34:59.805407 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:59.806041 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:34:59.806061 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:34:59.807875 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:34:59.810226 (+ 2351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:34:59.810230 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:34:59.810232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:34:59.810543 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:34:59.811085 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:34:59.811088 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:34:59.811090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:34:59.811246 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:34:59.811815 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:34:59.811825 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:34:59.811912 (+ 87us) client_negotiation.cc:769] Sending connection context
0929 18:34:59.812070 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 18:34:59.812229 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 18:34:59.812616 10493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:59.801354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33828 (local address 127.28.207.125:46659)
0929 18:34:59.801494 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 18:34:59.801501 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:34:59.801518 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:34:59.801563 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:34:59.801569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:34:59.801627 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:34:59.801712 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:34:59.802358 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.803248 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.804813 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:34:59.805079 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:34:59.805185 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:34:59.808077 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:34:59.808084 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:34:59.808089 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:34:59.808120 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:34:59.810025 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:59.810685 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:59.810689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:59.810690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:59.810773 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:34:59.811449 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:34:59.811452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:34:59.811454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:34:59.811696 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:34:59.811789 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 18:34:59.812210 (+ 421us) server_negotiation.cc:299] Negotiation successful
0929 18:34:59.812396 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 18:34:59.813674 10491 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.207.124" port: 32901 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:34:59.814740 10495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:59.814411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:32901 (local address 127.0.0.1:48612)
0929 18:34:59.814551 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:59.814657 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.124:32901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 18:34:59.815505 10441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.207.124:32901: connect: Connection refused (error 111)
I20260929 18:34:59.816706 10495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:59.816478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:32901 (local address 127.0.0.1:48616)
0929 18:34:59.816592 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:59.816647 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.124:32901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 18:34:59.816890 10491 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.207.124:32901: Network error: Client connection negotiation failed: client connection to 127.28.207.124:32901: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.124.keytab.
Entry for principal kudu/127.28.207.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.124.keytab.
I20260929 18:34:59.820436 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.124:32901
--webserver_interface=127.28.207.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.124.keytab
--principal=kudu/127.28.207.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:41571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:35695,127.28.207.125:46659,127.28.207.124:32901
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:34:59.834156 10416 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.207.124" port: 32901 } attempt: 1
I20260929 18:34:59.835387 10419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:59.834815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:32901 (local address 127.0.0.1:48622)
0929 18:34:59.835169 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:59.835271 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.124:32901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260929 18:34:59.836583 10419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:59.836210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:32901 (local address 127.0.0.1:48636)
0929 18:34:59.836348 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:59.836411 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.124:32901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260929 18:34:59.836826 10416 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.207.124:32901: Network error: Client connection negotiation failed: client connection to 127.28.207.124:32901: connect: Connection refused (error 111)
I20260929 18:34:59.866246 10491 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.207.124" port: 32901 } attempt: 1
I20260929 18:34:59.867455 10495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:59.867089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:32901 (local address 127.0.0.1:48638)
0929 18:34:59.867268 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:59.867347 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.124:32901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 18:34:59.868558 10495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:59.868285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:32901 (local address 127.0.0.1:48648)
0929 18:34:59.868450 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:59.868502 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.124:32901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260929 18:34:59.868927 10491 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.207.124:32901: Network error: Client connection negotiation failed: client connection to 127.28.207.124:32901: connect: Connection refused (error 111)
I20260929 18:34:59.912088 10416 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.207.124" port: 32901 } attempt: 2
I20260929 18:34:59.913226 10419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:59.912716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:32901 (local address 127.0.0.1:48656)
0929 18:34:59.913016 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:59.913114 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.124:32901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260929 18:34:59.914281 10419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:59.914011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:32901 (local address 127.0.0.1:48658)
0929 18:34:59.914159 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:59.914218 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.124:32901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260929 18:34:59.914552 10416 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.207.124:32901: Network error: Client connection negotiation failed: client connection to 127.28.207.124:32901: connect: Connection refused (error 111)
I20260929 18:34:59.937160 10491 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.207.124" port: 32901 } attempt: 2
I20260929 18:34:59.938424 10495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:59.937952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:32901 (local address 127.0.0.1:48668)
0929 18:34:59.938234 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:59.938323 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.124:32901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260929 18:34:59.939508 10495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:34:59.939277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:32901 (local address 127.0.0.1:48674)
0929 18:34:59.939387 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 18:34:59.939450 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.124:32901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 18:34:59.939634 10491 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.207.124:32901: Network error: Client connection negotiation failed: client connection to 127.28.207.124:32901: connect: Connection refused (error 111)
W20260929 18:34:59.941187 10498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:34:59.941521 10498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:34:59.941625 10498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:34:59.946796 10498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:34:59.946951 10498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:34:59.946996 10498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:34:59.947077 10498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:34:59.947122 10498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:34:59.953413 10498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:41571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:35695,127.28.207.125:46659,127.28.207.124:32901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.124.keytab
--principal=kudu/127.28.207.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.124:32901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.207.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:34:59.954628 10498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:34:59.955789 10498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:34:59.965590 10506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:59.965619 10504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:34:59.965615 10503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:34:59.966154 10498 server_base.cc:1034] running on GCE node
I20260929 18:34:59.966915 10498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:34:59.968593 10498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:34:59.969810 10498 hybrid_clock.cc:648] HybridClock initialized: now 1790706899969770 us; error 64 us; skew 500 ppm
Sep 29 18:34:59 dist-test-slave-qcpv krb5kdc[10335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706899, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:34:59.973274 10498 init.cc:377] Logged in from keytab as kudu/127.28.207.124@KRBTEST.COM (short username kudu)
I20260929 18:34:59.974788 10498 webserver.cc:460] Webserver started at http://127.28.207.124:40515/ using document root <none> and password file <none>
I20260929 18:34:59.975582 10498 fs_manager.cc:359] Metadata directory not provided
I20260929 18:34:59.975659 10498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:34:59.975932 10498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:34:59.978510 10498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-2/data/instance:
uuid: "26cb4537ee354f0cbda9924f774e7208"
format_stamp: "Formatted at 2026-09-29 18:34:59 on dist-test-slave-qcpv"
server_key: "1e6b3533e8a89393c30fa34d0edd7430"
server_key_iv: "dd052e9272a9b03c99a5466407c19519"
server_key_version: "encryptionkey@0"
I20260929 18:34:59.979255 10498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-2/wal/instance:
uuid: "26cb4537ee354f0cbda9924f774e7208"
format_stamp: "Formatted at 2026-09-29 18:34:59 on dist-test-slave-qcpv"
server_key: "1e6b3533e8a89393c30fa34d0edd7430"
server_key_iv: "dd052e9272a9b03c99a5466407c19519"
server_key_version: "encryptionkey@0"
I20260929 18:34:59.984094 10498 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260929 18:34:59.987766 10513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:34:59.989478 10498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:34:59.989674 10498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-2/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-2/wal
uuid: "26cb4537ee354f0cbda9924f774e7208"
format_stamp: "Formatted at 2026-09-29 18:34:59 on dist-test-slave-qcpv"
server_key: "1e6b3533e8a89393c30fa34d0edd7430"
server_key_iv: "dd052e9272a9b03c99a5466407c19519"
server_key_version: "encryptionkey@0"
I20260929 18:34:59.989825 10498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:00.011569 10498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:00.013779 10416 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.207.124" port: 32901 } attempt: 3
I20260929 18:35:00.015089 10419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:00.014537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:32901 (local address 127.0.0.1:48680)
0929 18:35:00.014837 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:00.014958 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.124:32901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
I20260929 18:35:00.016079 10419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:00.015793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:32901 (local address 127.0.0.1:48696)
0929 18:35:00.015941 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:00.016005 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.124:32901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 18:35:00.016352 10498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
W20260929 18:35:00.016623 10416 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.207.124:32901: Network error: Client connection negotiation failed: client connection to 127.28.207.124:32901: connect: Connection refused (error 111)
I20260929 18:35:00.016775 10498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:00.026741 10498 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.124:32901
I20260929 18:35:00.027112 10565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.124:32901 every 8 connection(s)
I20260929 18:35:00.028256 10498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-2/data/info.pb
I20260929 18:35:00.029423 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 10498
I20260929 18:35:00.029551 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790706718260899-29501-0/minicluster-data/master-2/wal/instance
I20260929 18:35:00.029881 29501 external_mini_cluster.cc:1428] Setting key 34411f19c282b9b9e925896724f75e1a
I20260929 18:35:00.030916 10491 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.207.124" port: 32901 } attempt: 3
I20260929 18:35:00.032555 10566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:00.037880 10566 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.207.126" port: 35695 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:35:00 dist-test-slave-qcpv krb5kdc[10335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706899, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.125@KRBTEST.COM for kudu/127.28.207.124@KRBTEST.COM
Sep 29 18:35:00 dist-test-slave-qcpv krb5kdc[10335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.207.124@KRBTEST.COM
I20260929 18:35:00.047959 10495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:00.031503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:32901 (local address 127.0.0.1:48720)
0929 18:35:00.031674 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:00.031694 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:00.031809 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:00.034461 (+ 2652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:00.034464 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:00.034969 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:00.035166 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:00.035178 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:00.037250 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:00.037258 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:00.038076 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:00.038091 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:00.039492 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:00.039496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:00.039597 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:00.040337 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:00.040352 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:00.042354 (+ 2002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:00.045378 (+ 3024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:00.045382 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:00.045385 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:00.045709 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:00.046307 (+ 598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:00.046311 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:00.046313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:00.046421 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:00.047273 (+ 852us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:00.047277 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:00.047412 (+ 135us) client_negotiation.cc:769] Sending connection context
0929 18:35:00.047533 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 18:35:00.047729 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
Sep 29 18:35:00 dist-test-slave-qcpv krb5kdc[10335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706899, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.124@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:35:00.049575 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:00.031617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48708 (local address 127.28.207.124:32901)
0929 18:35:00.032210 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:00.032238 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:00.032613 (+ 375us) server_negotiation.cc:407] Connection header received
0929 18:35:00.033532 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:00.033539 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:00.034011 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:00.034434 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:00.035764 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:00.036996 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:00.038198 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:00.039333 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:00.039613 (+ 280us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:00.044626 (+ 5013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:00.044630 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:00.044633 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:00.044663 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:00.047156 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:00.047815 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:00.047819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:00.047821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:00.047925 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:00.048235 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:00.048238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:00.048240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:00.048493 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:00.048674 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:00.049054 (+ 380us) server_negotiation.cc:299] Negotiation successful
0929 18:35:00.049190 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":157,"server-negotiator.queue_time_us":377,"spinlock_wait_cycles":7040,"thread_start_us":187,"threads_started":1}
I20260929 18:35:00.049598 10570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:00.032313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48720 (local address 127.28.207.124:32901)
0929 18:35:00.032603 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:00.032612 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:00.032626 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:35:00.033496 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:00.033519 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:00.034012 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:00.034952 (+ 940us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:00.035350 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:00.037024 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:00.038288 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:00.039333 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:00.039611 (+ 278us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:00.042626 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:00.042638 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:00.042659 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:00.042705 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:00.045200 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:00.045879 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:00.045887 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:00.045905 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:00.046134 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:00.046533 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:00.046538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:00.046540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:00.047149 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:00.047309 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:00.047736 (+ 427us) server_negotiation.cc:299] Negotiation successful
0929 18:35:00.048162 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":584,"server-negotiator.queue_time_us":218,"spinlock_wait_cycles":17152,"thread_start_us":150,"threads_started":1}
I20260929 18:35:00.053347 10420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:00.041128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57574 (local address 127.28.207.126:35695)
0929 18:35:00.041369 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:00.041377 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:00.041394 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:35:00.041444 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:00.041450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:00.041511 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:00.041621 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:00.042483 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:00.043175 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:00.044750 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:00.045112 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:00.045228 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:00.048920 (+ 3692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:00.048925 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:00.048928 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:00.048969 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:00.051264 (+ 2295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:00.051881 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:00.051886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:00.051888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:00.051992 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:00.052319 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:00.052324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:00.052328 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:00.052551 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:00.052611 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:00.053020 (+ 409us) server_negotiation.cc:299] Negotiation successful
0929 18:35:00.053160 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260929 18:35:00.053983 10571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:00.039989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:35695 (local address 127.0.0.1:57574)
0929 18:35:00.040579 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:00.040633 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:00.040747 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:00.041681 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:00.041688 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:00.042206 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:00.042347 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:00.042353 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:00.043342 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:00.043349 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:00.044594 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:00.044607 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:00.045243 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:00.045248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:00.045417 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:00.045970 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:00.045989 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:00.048716 (+ 2727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:00.051398 (+ 2682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:00.051404 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:00.051408 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:00.051735 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:00.052112 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:00.052115 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:00.052116 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:00.052198 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:00.052679 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:00.052683 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:00.052780 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 18:35:00.053589 (+ 809us) client_negotiation.cc:240] Negotiation successful
0929 18:35:00.053808 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":140,"threads_started":1}
I20260929 18:35:00.055634 10491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 648eb2f09f534d12b84bb37df0882df7: Bootstrap starting.
I20260929 18:35:00.056143 10566 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.207.125" port: 46659 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:35:00 dist-test-slave-qcpv krb5kdc[10335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:35:00.059854 10491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 648eb2f09f534d12b84bb37df0882df7: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:00.061118 10491 log.cc:824] T 00000000000000000000000000000000 P 648eb2f09f534d12b84bb37df0882df7: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:35:00 dist-test-slave-qcpv krb5kdc[10335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706899, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.124@KRBTEST.COM for kudu/127.28.207.125@KRBTEST.COM
I20260929 18:35:00.064330 10491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 648eb2f09f534d12b84bb37df0882df7: No bootstrap required, opened a new log
I20260929 18:35:00.064865 10420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:00.053260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57584 (local address 127.28.207.126:35695)
0929 18:35:00.053543 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:00.053546 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:00.053559 (+ 13us) server_negotiation.cc:407] Connection header received
0929 18:35:00.053799 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:00.053803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:00.053865 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:00.053986 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:00.055072 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:00.055668 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:00.056644 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:00.056947 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:00.057050 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:00.060259 (+ 3209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:00.060264 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:00.060267 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:00.060302 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:00.062441 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:00.063072 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:00.063077 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:00.063079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:00.063183 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:00.063758 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:00.063762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:00.063764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:00.064047 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:00.064333 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:00.064487 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 18:35:00.064637 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":95,"threads_started":1}
I20260929 18:35:00.067075 10493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:00.057002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33836 (local address 127.28.207.125:46659)
0929 18:35:00.057155 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:00.057161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:00.057177 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:35:00.057222 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:00.057225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:00.057275 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:00.057364 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:00.057913 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:00.058831 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:00.059975 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:00.060320 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:00.060459 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:00.062910 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:00.062914 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:00.062917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:00.062945 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:00.064761 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:00.065405 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:00.065409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:00.065411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:00.065523 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:00.065910 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:00.065916 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:00.065917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:00.066215 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:00.066330 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:00.066699 (+ 369us) server_negotiation.cc:299] Negotiation successful
0929 18:35:00.066904 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 18:35:00.067806 10571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:00.056748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.125:46659 (local address 127.0.0.1:33836)
0929 18:35:00.056889 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:00.056905 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:00.056983 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:00.057418 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:00.057422 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:00.057630 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:00.057757 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:00.057766 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:00.059020 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:00.059024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:00.059814 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:00.059824 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:00.060488 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:00.060493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:00.060604 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:00.061178 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:00.061193 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:00.062700 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:00.064908 (+ 2208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:00.064912 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:00.064914 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:00.065173 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:00.065679 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:00.065683 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:00.065685 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:00.065801 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:00.066340 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:00.066344 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:00.066487 (+ 143us) client_negotiation.cc:769] Sending connection context
0929 18:35:00.067290 (+ 803us) client_negotiation.cc:240] Negotiation successful
0929 18:35:00.067612 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 18:35:00.068747 10566 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.207.124" port: 32901 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:35:00.070019 10491 raft_consensus.cc:348] T 00000000000000000000000000000000 P 648eb2f09f534d12b84bb37df0882df7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d82a3df62548b0b732a58cf70784c8" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 35695 } } peers { permanent_uuid: "648eb2f09f534d12b84bb37df0882df7" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 46659 } } peers { permanent_uuid: "26cb4537ee354f0cbda9924f774e7208" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 32901 } }
I20260929 18:35:00.070341 10491 raft_consensus.cc:374] T 00000000000000000000000000000000 P 648eb2f09f534d12b84bb37df0882df7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:00.070451 10491 raft_consensus.cc:729] T 00000000000000000000000000000000 P 648eb2f09f534d12b84bb37df0882df7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 648eb2f09f534d12b84bb37df0882df7, State: Initialized, Role: FOLLOWER
I20260929 18:35:00.071087 10491 consensus_queue.cc:260] T 00000000000000000000000000000000 P 648eb2f09f534d12b84bb37df0882df7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d82a3df62548b0b732a58cf70784c8" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 35695 } } peers { permanent_uuid: "648eb2f09f534d12b84bb37df0882df7" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 46659 } } peers { permanent_uuid: "26cb4537ee354f0cbda9924f774e7208" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 32901 } }
Sep 29 18:35:00 dist-test-slave-qcpv krb5kdc[10335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.207.125@KRBTEST.COM
I20260929 18:35:00.072956 10491 sys_catalog.cc:564] T 00000000000000000000000000000000 P 648eb2f09f534d12b84bb37df0882df7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:35:00.073109 10575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 648eb2f09f534d12b84bb37df0882df7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d82a3df62548b0b732a58cf70784c8" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 35695 } } peers { permanent_uuid: "648eb2f09f534d12b84bb37df0882df7" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 46659 } } peers { permanent_uuid: "26cb4537ee354f0cbda9924f774e7208" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 32901 } } }
I20260929 18:35:00.074520 10575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 648eb2f09f534d12b84bb37df0882df7 [sys.catalog]: This master's current role is: FOLLOWER
Sep 29 18:35:00 dist-test-slave-qcpv krb5kdc[10335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706899, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.124@KRBTEST.COM for kudu/127.28.207.124@KRBTEST.COM
I20260929 18:35:00.079824 10571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:00.069296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:32901 (local address 127.0.0.1:48734)
0929 18:35:00.069459 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:00.069470 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:00.069550 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:00.069763 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:00.069766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:00.069985 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:00.070065 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:00.070071 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:00.071146 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:00.071149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:00.071903 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:00.071914 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:00.072587 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:00.072591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:00.072658 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:00.073190 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:00.073204 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:00.075325 (+ 2121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:00.077862 (+ 2537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:00.077866 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:00.077868 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:00.078198 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:00.078546 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:00.078552 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:00.078555 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:00.078633 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:00.079233 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:00.079236 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:00.079320 (+ 84us) client_negotiation.cc:769] Sending connection context
0929 18:35:00.079431 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 18:35:00.079593 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260929 18:35:00.080364 10570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:00.069325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48734 (local address 127.28.207.124:32901)
0929 18:35:00.069428 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:00.069431 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:00.069510 (+ 79us) server_negotiation.cc:407] Connection header received
0929 18:35:00.069626 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:00.069628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:00.069667 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:00.069753 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:00.070200 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:00.071015 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:00.072024 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:00.072341 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:00.072417 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:00.075528 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:00.075534 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:00.075539 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:00.075569 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:00.077698 (+ 2129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:00.078332 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:00.078336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:00.078338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:00.078434 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:00.078737 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:00.078740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:00.078742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:00.079080 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:00.079197 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:00.080065 (+ 868us) server_negotiation.cc:299] Negotiation successful
0929 18:35:00.080219 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 18:35:00.080837 10574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:00.066620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33838 (local address 127.28.207.125:46659)
0929 18:35:00.067055 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:00.067063 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:00.067073 (+ 10us) server_negotiation.cc:407] Connection header received
0929 18:35:00.067100 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:00.067102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:00.067188 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:00.067282 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:00.068043 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:00.068816 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:00.069978 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:00.070335 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:00.070432 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:00.073302 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:00.073306 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:00.073308 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:00.073344 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:00.077195 (+ 3851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:00.078043 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:00.078047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:00.078049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:00.078150 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:00.078499 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:00.078505 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:00.078507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:00.078801 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:00.079112 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:00.080463 (+ 1351us) server_negotiation.cc:299] Negotiation successful
0929 18:35:00.080620 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":146,"threads_started":1}
W20260929 18:35:00.081986 10586 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 648eb2f09f534d12b84bb37df0882df7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:35:00.082190 10586 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:35:00.078758 10423 master_runner.cc:186] Error getting master registration for 127.28.207.126:35695: OK, instance_id { permanent_uuid: "d8d82a3df62548b0b732a58cf70784c8" instance_seqno: 1790706899525007 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 18:35:00.083899 10423 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:35:00.084239 10566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26cb4537ee354f0cbda9924f774e7208: Bootstrap starting.
I20260929 18:35:00.086127 29501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:35:00.087716 10566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26cb4537ee354f0cbda9924f774e7208: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:00.088706 10566 log.cc:824] T 00000000000000000000000000000000 P 26cb4537ee354f0cbda9924f774e7208: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:00.091342 10566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26cb4537ee354f0cbda9924f774e7208: No bootstrap required, opened a new log
I20260929 18:35:00.096757 10566 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26cb4537ee354f0cbda9924f774e7208 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d82a3df62548b0b732a58cf70784c8" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 35695 } } peers { permanent_uuid: "648eb2f09f534d12b84bb37df0882df7" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 46659 } } peers { permanent_uuid: "26cb4537ee354f0cbda9924f774e7208" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 32901 } }
I20260929 18:35:00.097219 10566 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26cb4537ee354f0cbda9924f774e7208 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:00.097330 10566 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26cb4537ee354f0cbda9924f774e7208 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26cb4537ee354f0cbda9924f774e7208, State: Initialized, Role: FOLLOWER
I20260929 18:35:00.098083 10566 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26cb4537ee354f0cbda9924f774e7208 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d82a3df62548b0b732a58cf70784c8" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 35695 } } peers { permanent_uuid: "648eb2f09f534d12b84bb37df0882df7" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 46659 } } peers { permanent_uuid: "26cb4537ee354f0cbda9924f774e7208" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 32901 } }
I20260929 18:35:00.100699 10566 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26cb4537ee354f0cbda9924f774e7208 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:35:00.101744 10596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26cb4537ee354f0cbda9924f774e7208 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d82a3df62548b0b732a58cf70784c8" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 35695 } } peers { permanent_uuid: "648eb2f09f534d12b84bb37df0882df7" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 46659 } } peers { permanent_uuid: "26cb4537ee354f0cbda9924f774e7208" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 32901 } } }
I20260929 18:35:00.103338 10420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:00.090410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57586 (local address 127.28.207.126:35695)
0929 18:35:00.090598 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:00.090604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:00.090621 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:35:00.090670 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:00.090673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:00.090726 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:00.090810 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:00.092362 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:00.093321 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:00.094746 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:00.095185 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:00.095303 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:00.098226 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:00.098232 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:00.098235 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:00.098272 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:00.100946 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:00.101534 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:00.101538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:00.101541 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:00.101647 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:00.102024 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:00.102028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:00.102030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:00.102333 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:00.102452 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:00.102964 (+ 512us) server_negotiation.cc:299] Negotiation successful
0929 18:35:00.103108 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260929 18:35:00.109517 10607 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 26cb4537ee354f0cbda9924f774e7208: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:35:00.109730 10607 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:35:00.107187 10498 master_runner.cc:186] Error getting master registration for 127.28.207.126:35695: OK, instance_id { permanent_uuid: "d8d82a3df62548b0b732a58cf70784c8" instance_seqno: 1790706899525007 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 18:35:00.111919 10570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:00.090578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48744 (local address 127.28.207.124:32901)
0929 18:35:00.090749 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:00.090757 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:00.090777 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:35:00.090844 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:00.090848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:00.090956 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:00.091054 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:00.098649 (+ 7595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:00.099464 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:00.105495 (+ 6031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:00.105805 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:00.105893 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:00.107894 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:00.107899 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:00.107902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:00.107931 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:00.109856 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:00.110375 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:00.110379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:00.110381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:00.110477 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:00.110801 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:00.110804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:00.110806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:00.111125 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:00.111322 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:00.111581 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 18:35:00.111712 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 18:35:00.112541 10498 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:35:00.112627 10596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26cb4537ee354f0cbda9924f774e7208 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:35:00.113386 10574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:00.091518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33854 (local address 127.28.207.125:46659)
0929 18:35:00.091711 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:00.091716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:00.094470 (+ 2754us) server_negotiation.cc:407] Connection header received
0929 18:35:00.094692 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:00.094712 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:00.094778 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:00.094840 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:00.096258 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:00.097127 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:00.098608 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:00.099029 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:00.099117 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:00.101263 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:00.101268 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:00.101271 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:00.101302 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:00.103442 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:00.104105 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:00.104109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:00.104111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:00.104213 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:00.111817 (+ 7604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:00.111825 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:00.111828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:00.112155 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:00.112326 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:00.113012 (+ 686us) server_negotiation.cc:299] Negotiation successful
0929 18:35:00.113240 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260929 18:35:00.127346 10575 raft_consensus.cc:482] T 00000000000000000000000000000000 P 648eb2f09f534d12b84bb37df0882df7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:35:00.127552 10575 raft_consensus.cc:504] T 00000000000000000000000000000000 P 648eb2f09f534d12b84bb37df0882df7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d82a3df62548b0b732a58cf70784c8" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 35695 } } peers { permanent_uuid: "648eb2f09f534d12b84bb37df0882df7" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 46659 } } peers { permanent_uuid: "26cb4537ee354f0cbda9924f774e7208" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 32901 } }
I20260929 18:35:00.128615 10575 leader_election.cc:288] T 00000000000000000000000000000000 P 648eb2f09f534d12b84bb37df0882df7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8d82a3df62548b0b732a58cf70784c8 (127.28.207.126:35695), 26cb4537ee354f0cbda9924f774e7208 (127.28.207.124:32901)
I20260929 18:35:00.129243 10540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "648eb2f09f534d12b84bb37df0882df7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26cb4537ee354f0cbda9924f774e7208" is_pre_election: true
I20260929 18:35:00.129659 10540 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 26cb4537ee354f0cbda9924f774e7208 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 648eb2f09f534d12b84bb37df0882df7 in term 0.
I20260929 18:35:00.130115 10441 leader_election.cc:302] T 00000000000000000000000000000000 P 648eb2f09f534d12b84bb37df0882df7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26cb4537ee354f0cbda9924f774e7208, 648eb2f09f534d12b84bb37df0882df7; no voters:
I20260929 18:35:00.130404 10575 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 648eb2f09f534d12b84bb37df0882df7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:35:00.130507 10575 raft_consensus.cc:482] T 00000000000000000000000000000000 P 648eb2f09f534d12b84bb37df0882df7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:35:00.130561 10575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 648eb2f09f534d12b84bb37df0882df7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:00.132597 10575 raft_consensus.cc:504] T 00000000000000000000000000000000 P 648eb2f09f534d12b84bb37df0882df7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d82a3df62548b0b732a58cf70784c8" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 35695 } } peers { permanent_uuid: "648eb2f09f534d12b84bb37df0882df7" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 46659 } } peers { permanent_uuid: "26cb4537ee354f0cbda9924f774e7208" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 32901 } }
I20260929 18:35:00.134295 10540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "648eb2f09f534d12b84bb37df0882df7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26cb4537ee354f0cbda9924f774e7208"
I20260929 18:35:00.134425 10540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26cb4537ee354f0cbda9924f774e7208 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:00.135823 10540 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 26cb4537ee354f0cbda9924f774e7208 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 648eb2f09f534d12b84bb37df0882df7 in term 1.
I20260929 18:35:00.136075 10575 leader_election.cc:288] T 00000000000000000000000000000000 P 648eb2f09f534d12b84bb37df0882df7 [CANDIDATE]: Term 1 election: Requested vote from peers d8d82a3df62548b0b732a58cf70784c8 (127.28.207.126:35695), 26cb4537ee354f0cbda9924f774e7208 (127.28.207.124:32901)
I20260929 18:35:00.136114 10441 leader_election.cc:302] T 00000000000000000000000000000000 P 648eb2f09f534d12b84bb37df0882df7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26cb4537ee354f0cbda9924f774e7208, 648eb2f09f534d12b84bb37df0882df7; no voters:
I20260929 18:35:00.136334 10575 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 648eb2f09f534d12b84bb37df0882df7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:00.136574 10575 raft_consensus.cc:686] T 00000000000000000000000000000000 P 648eb2f09f534d12b84bb37df0882df7 [term 1 LEADER]: Becoming Leader. State: Replica: 648eb2f09f534d12b84bb37df0882df7, State: Running, Role: LEADER
I20260929 18:35:00.136958 10575 consensus_queue.cc:237] T 00000000000000000000000000000000 P 648eb2f09f534d12b84bb37df0882df7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d82a3df62548b0b732a58cf70784c8" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 35695 } } peers { permanent_uuid: "648eb2f09f534d12b84bb37df0882df7" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 46659 } } peers { permanent_uuid: "26cb4537ee354f0cbda9924f774e7208" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 32901 } }
I20260929 18:35:00.140712 10609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 648eb2f09f534d12b84bb37df0882df7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 648eb2f09f534d12b84bb37df0882df7. Latest consensus state: current_term: 1 leader_uuid: "648eb2f09f534d12b84bb37df0882df7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d82a3df62548b0b732a58cf70784c8" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 35695 } } peers { permanent_uuid: "648eb2f09f534d12b84bb37df0882df7" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 46659 } } peers { permanent_uuid: "26cb4537ee354f0cbda9924f774e7208" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 32901 } } }
I20260929 18:35:00.140837 10609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 648eb2f09f534d12b84bb37df0882df7 [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:00.141307 10611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:35:00.143013 10611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:35:00.149571 10540 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 26cb4537ee354f0cbda9924f774e7208 [term 1 FOLLOWER]: Refusing update from remote peer 648eb2f09f534d12b84bb37df0882df7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:35:00.150265 10609 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 648eb2f09f534d12b84bb37df0882df7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26cb4537ee354f0cbda9924f774e7208" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 32901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:35:00.154850 10596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26cb4537ee354f0cbda9924f774e7208 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 648eb2f09f534d12b84bb37df0882df7. Latest consensus state: current_term: 1 leader_uuid: "648eb2f09f534d12b84bb37df0882df7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d82a3df62548b0b732a58cf70784c8" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 35695 } } peers { permanent_uuid: "648eb2f09f534d12b84bb37df0882df7" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 46659 } } peers { permanent_uuid: "26cb4537ee354f0cbda9924f774e7208" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 32901 } } }
I20260929 18:35:00.155011 10596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26cb4537ee354f0cbda9924f774e7208 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:35:00.159060 10575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 648eb2f09f534d12b84bb37df0882df7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "648eb2f09f534d12b84bb37df0882df7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d82a3df62548b0b732a58cf70784c8" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 35695 } } peers { permanent_uuid: "648eb2f09f534d12b84bb37df0882df7" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 46659 } } peers { permanent_uuid: "26cb4537ee354f0cbda9924f774e7208" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 32901 } } }
I20260929 18:35:00.159194 10575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 648eb2f09f534d12b84bb37df0882df7 [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:00.159745 10612 mvcc.cc:205] Tried to move back new op lower bound from 7334735463002300416 to 7334735462964482048. Current Snapshot: MvccSnapshot[applied={T|T < 7334735463002300416}]
I20260929 18:35:00.160589 10596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26cb4537ee354f0cbda9924f774e7208 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "648eb2f09f534d12b84bb37df0882df7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d82a3df62548b0b732a58cf70784c8" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 35695 } } peers { permanent_uuid: "648eb2f09f534d12b84bb37df0882df7" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 46659 } } peers { permanent_uuid: "26cb4537ee354f0cbda9924f774e7208" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 32901 } } }
I20260929 18:35:00.160724 10596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26cb4537ee354f0cbda9924f774e7208 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:35:00.160892 10613 mvcc.cc:205] Tried to move back new op lower bound from 7334735463002300416 to 7334735462964482048. Current Snapshot: MvccSnapshot[applied={T|T < 7334735463002300416}]
I20260929 18:35:00.163774 10611 catalog_manager.cc:1347] Generated new cluster ID: b126971e64a24310888eca64f726ea4d
I20260929 18:35:00.163837 10611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:35:00.180816 10416 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.207.124" port: 32901 } attempt: 4
I20260929 18:35:00.181790 10611 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:35:00.183768 10611 catalog_manager.cc:1504] Loading token signing keys...
Sep 29 18:35:00 dist-test-slave-qcpv krb5kdc[10335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706899, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for kudu/127.28.207.124@KRBTEST.COM
I20260929 18:35:00.191954 10419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:00.181399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:32901 (local address 127.0.0.1:48748)
0929 18:35:00.181618 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:00.181635 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:00.181741 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:00.182177 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:00.182181 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:00.182447 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:00.182587 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:00.182595 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:00.183687 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:00.183690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:00.184438 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:00.184451 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:00.185381 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:00.185384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:00.185476 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:00.186142 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:00.186157 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:00.187911 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:00.190205 (+ 2294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:00.190209 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:00.190211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:00.190442 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:00.190762 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:00.190765 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:00.190767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:00.190875 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:00.191373 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:00.191376 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:00.191493 (+ 117us) client_negotiation.cc:769] Sending connection context
0929 18:35:00.191581 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 18:35:00.191736 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260929 18:35:00.192953 10570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:00.181440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48748 (local address 127.28.207.124:32901)
0929 18:35:00.181575 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:00.181581 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:00.181691 (+ 110us) server_negotiation.cc:407] Connection header received
0929 18:35:00.181842 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:00.181845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:00.181923 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:00.182012 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:00.182729 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:00.183513 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:00.184623 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:00.185073 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:00.185192 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:00.188105 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:00.188109 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:00.188113 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:00.188140 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:00.190040 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:00.190564 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:00.190567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:00.190586 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:00.190660 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:00.191001 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:00.191004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:00.191006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:00.191238 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:00.191315 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:00.192353 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0929 18:35:00.192620 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 18:35:00.197376 10416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8d82a3df62548b0b732a58cf70784c8: Bootstrap starting.
I20260929 18:35:00.202529 10611 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 648eb2f09f534d12b84bb37df0882df7: Generated new TSK 0
I20260929 18:35:00.205099 10611 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:35:00.207406 10416 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8d82a3df62548b0b732a58cf70784c8: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:00.208581 10416 log.cc:824] T 00000000000000000000000000000000 P d8d82a3df62548b0b732a58cf70784c8: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:00.212879 10416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8d82a3df62548b0b732a58cf70784c8: No bootstrap required, opened a new log
I20260929 18:35:00.217978 10416 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8d82a3df62548b0b732a58cf70784c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d82a3df62548b0b732a58cf70784c8" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 35695 } } peers { permanent_uuid: "648eb2f09f534d12b84bb37df0882df7" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 46659 } } peers { permanent_uuid: "26cb4537ee354f0cbda9924f774e7208" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 32901 } }
I20260929 18:35:00.218317 10416 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8d82a3df62548b0b732a58cf70784c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:00.218385 10416 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8d82a3df62548b0b732a58cf70784c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d82a3df62548b0b732a58cf70784c8, State: Initialized, Role: FOLLOWER
I20260929 18:35:00.219069 10416 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8d82a3df62548b0b732a58cf70784c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d82a3df62548b0b732a58cf70784c8" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 35695 } } peers { permanent_uuid: "648eb2f09f534d12b84bb37df0882df7" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 46659 } } peers { permanent_uuid: "26cb4537ee354f0cbda9924f774e7208" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 32901 } }
I20260929 18:35:00.220965 10416 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8d82a3df62548b0b732a58cf70784c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:35:00.221127 10389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "648eb2f09f534d12b84bb37df0882df7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d82a3df62548b0b732a58cf70784c8"
I20260929 18:35:00.221390 10389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8d82a3df62548b0b732a58cf70784c8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:00.223304 10389 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d8d82a3df62548b0b732a58cf70784c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 648eb2f09f534d12b84bb37df0882df7 in term 1.
I20260929 18:35:00.224502 10388 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d8d82a3df62548b0b732a58cf70784c8 [term 1 FOLLOWER]: Refusing update from remote peer 648eb2f09f534d12b84bb37df0882df7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:35:00.224960 10618 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8d82a3df62548b0b732a58cf70784c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d82a3df62548b0b732a58cf70784c8" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 35695 } } peers { permanent_uuid: "648eb2f09f534d12b84bb37df0882df7" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 46659 } } peers { permanent_uuid: "26cb4537ee354f0cbda9924f774e7208" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 32901 } } }
I20260929 18:35:00.225168 10618 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8d82a3df62548b0b732a58cf70784c8 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:35:00.225283 10390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "648eb2f09f534d12b84bb37df0882df7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8d82a3df62548b0b732a58cf70784c8" is_pre_election: true
I20260929 18:35:00.225420 10390 raft_consensus.cc:2362] T 00000000000000000000000000000000 P d8d82a3df62548b0b732a58cf70784c8 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 648eb2f09f534d12b84bb37df0882df7 in term 1. Re-sending same reply.
I20260929 18:35:00.226509 10609 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 648eb2f09f534d12b84bb37df0882df7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d82a3df62548b0b732a58cf70784c8" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 35695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 18:35:00.230723 10630 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d8d82a3df62548b0b732a58cf70784c8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:35:00.230913 10630 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:35:00.233536 10618 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8d82a3df62548b0b732a58cf70784c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 648eb2f09f534d12b84bb37df0882df7. Latest consensus state: current_term: 1 leader_uuid: "648eb2f09f534d12b84bb37df0882df7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d82a3df62548b0b732a58cf70784c8" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 35695 } } peers { permanent_uuid: "648eb2f09f534d12b84bb37df0882df7" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 46659 } } peers { permanent_uuid: "26cb4537ee354f0cbda9924f774e7208" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 32901 } } }
I20260929 18:35:00.233671 10618 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8d82a3df62548b0b732a58cf70784c8 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:35:00.233758 10618 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8d82a3df62548b0b732a58cf70784c8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "648eb2f09f534d12b84bb37df0882df7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d82a3df62548b0b732a58cf70784c8" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 35695 } } peers { permanent_uuid: "648eb2f09f534d12b84bb37df0882df7" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 46659 } } peers { permanent_uuid: "26cb4537ee354f0cbda9924f774e7208" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 32901 } } }
I20260929 18:35:00.233817 10618 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8d82a3df62548b0b732a58cf70784c8 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:35:00.242343 10633 mvcc.cc:205] Tried to move back new op lower bound from 7334735463201665024 to 7334735462964482048. Current Snapshot: MvccSnapshot[applied={T|T < 7334735463002300416}]
I20260929 18:35:00.607159 10609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 648eb2f09f534d12b84bb37df0882df7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "648eb2f09f534d12b84bb37df0882df7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8d82a3df62548b0b732a58cf70784c8" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 35695 } } peers { permanent_uuid: "648eb2f09f534d12b84bb37df0882df7" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 46659 } } peers { permanent_uuid: "26cb4537ee354f0cbda9924f774e7208" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 32901 } } }
I20260929 18:35:00.607438 10609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 648eb2f09f534d12b84bb37df0882df7 [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:01.111595 10607 catalog_manager.cc:1259] Loaded cluster ID: b126971e64a24310888eca64f726ea4d
I20260929 18:35:01.111724 10607 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 26cb4537ee354f0cbda9924f774e7208: loading cluster ID for follower catalog manager: success
I20260929 18:35:01.113621 10607 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 26cb4537ee354f0cbda9924f774e7208: acquiring CA information for follower catalog manager: success
I20260929 18:35:01.115033 10607 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 26cb4537ee354f0cbda9924f774e7208: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:35:01.232378 10630 catalog_manager.cc:1259] Loaded cluster ID: b126971e64a24310888eca64f726ea4d
I20260929 18:35:01.232503 10630 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d8d82a3df62548b0b732a58cf70784c8: loading cluster ID for follower catalog manager: success
I20260929 18:35:01.234266 10630 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d8d82a3df62548b0b732a58cf70784c8: acquiring CA information for follower catalog manager: success
I20260929 18:35:01.235590 10630 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d8d82a3df62548b0b732a58cf70784c8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:35:01.297050 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 10348
I20260929 18:35:01.306165 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 10423
I20260929 18:35:01.313726 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 10498
2026-09-29T18:35:01Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3278 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:35:01 dist-test-slave-qcpv krb5kdc[10643](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:35:01 dist-test-slave-qcpv krb5kdc[10643](info): set up 2 sockets
Sep 29 18:35:01 dist-test-slave-qcpv krb5kdc[10643](info): commencing operation
krb5kdc: starting...
W20260929 18:35:02.387188 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:35:02 dist-test-slave-qcpv krb5kdc[10643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:35:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:35:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.207.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
I20260929 18:35:02.530323 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:40053
--webserver_interface=127.28.207.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:41071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:40053
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:02.644704 10656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:02.645004 10656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:02.645046 10656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:02.649238 10656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:35:02.649312 10656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:02.649333 10656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:02.649349 10656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:35:02.649366 10656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:35:02.654795 10656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:41071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:40053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:40053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:35:02.655954 10656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:02.656998 10656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:02.666543 10661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:02.666539 10664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:02.666664 10662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:02.667191 10656 server_base.cc:1034] running on GCE node
I20260929 18:35:02.667851 10656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:02.669102 10656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:02.670336 10656 hybrid_clock.cc:648] HybridClock initialized: now 1790706902670321 us; error 36 us; skew 500 ppm
Sep 29 18:35:02 dist-test-slave-qcpv krb5kdc[10643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706902, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:02.673372 10656 init.cc:377] Logged in from keytab as kudu/127.28.207.126@KRBTEST.COM (short username kudu)
I20260929 18:35:02.674391 10656 webserver.cc:460] Webserver started at http://127.28.207.126:44341/ using document root <none> and password file <none>
I20260929 18:35:02.675001 10656 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:02.675072 10656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:02.675253 10656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:02.677352 10656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "19f08364540043328e9da20b4678d849"
format_stamp: "Formatted at 2026-09-29 18:35:02 on dist-test-slave-qcpv"
server_key: "5c034467881caf83bfefb28590a5907b"
server_key_iv: "674fbe71c93d8dd0458f5d389390df17"
server_key_version: "encryptionkey@0"
I20260929 18:35:02.677973 10656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "19f08364540043328e9da20b4678d849"
format_stamp: "Formatted at 2026-09-29 18:35:02 on dist-test-slave-qcpv"
server_key: "5c034467881caf83bfefb28590a5907b"
server_key_iv: "674fbe71c93d8dd0458f5d389390df17"
server_key_version: "encryptionkey@0"
I20260929 18:35:02.682163 10656 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:35:02.685021 10671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:02.686568 10656 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 18:35:02.686729 10656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "19f08364540043328e9da20b4678d849"
format_stamp: "Formatted at 2026-09-29 18:35:02 on dist-test-slave-qcpv"
server_key: "5c034467881caf83bfefb28590a5907b"
server_key_iv: "674fbe71c93d8dd0458f5d389390df17"
server_key_version: "encryptionkey@0"
I20260929 18:35:02.686857 10656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:02.710963 10656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:02.714490 10656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:02.714761 10656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:02.723409 10656 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:40053
I20260929 18:35:02.723444 10723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:40053 every 8 connection(s)
I20260929 18:35:02.724560 10656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:35:02.728252 10724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:02.728811 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 10656
I20260929 18:35:02.728922 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:35:02.729259 29501 external_mini_cluster.cc:1428] Setting key 76296e4da23685a995c598afba8fba51
I20260929 18:35:02.737538 10724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19f08364540043328e9da20b4678d849: Bootstrap starting.
Sep 29 18:35:02 dist-test-slave-qcpv krb5kdc[10643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:35:02.741084 10724 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19f08364540043328e9da20b4678d849: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:02.742053 10724 log.cc:824] T 00000000000000000000000000000000 P 19f08364540043328e9da20b4678d849: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:02.745133 10724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19f08364540043328e9da20b4678d849: No bootstrap required, opened a new log
I20260929 18:35:02.748330 10727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:02.731064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51320 (local address 127.28.207.126:40053)
0929 18:35:02.731696 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:02.731714 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:02.731758 (+ 44us) server_negotiation.cc:407] Connection header received
0929 18:35:02.732678 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:02.732699 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:02.733074 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:02.733517 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:02.734704 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:02.735872 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:02.737294 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:02.738000 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:02.738147 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:02.741532 (+ 3385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:02.741538 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:02.741555 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:02.741586 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:02.743860 (+ 2274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:02.744794 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:02.744799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:02.744805 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:02.744958 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:02.745355 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:02.745360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:02.745363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:02.746004 (+ 641us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:02.746236 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:02.746627 (+ 391us) server_negotiation.cc:299] Negotiation successful
0929 18:35:02.747042 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":228,"threads_started":1}
I20260929 18:35:02.750602 10724 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19f08364540043328e9da20b4678d849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19f08364540043328e9da20b4678d849" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 40053 } }
I20260929 18:35:02.751008 10724 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19f08364540043328e9da20b4678d849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:02.751122 10724 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19f08364540043328e9da20b4678d849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f08364540043328e9da20b4678d849, State: Initialized, Role: FOLLOWER
I20260929 18:35:02.751685 10724 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19f08364540043328e9da20b4678d849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19f08364540043328e9da20b4678d849" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 40053 } }
I20260929 18:35:02.751875 10724 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19f08364540043328e9da20b4678d849 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:35:02.751974 10724 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19f08364540043328e9da20b4678d849 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:35:02.752095 10724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19f08364540043328e9da20b4678d849 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:02.752399 29501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:35:02.753423 10724 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19f08364540043328e9da20b4678d849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19f08364540043328e9da20b4678d849" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 40053 } }
I20260929 18:35:02.753870 10724 leader_election.cc:302] T 00000000000000000000000000000000 P 19f08364540043328e9da20b4678d849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19f08364540043328e9da20b4678d849; no voters:
I20260929 18:35:02.754323 10724 leader_election.cc:288] T 00000000000000000000000000000000 P 19f08364540043328e9da20b4678d849 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:35:02.754508 10729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19f08364540043328e9da20b4678d849 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:02.754834 10729 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19f08364540043328e9da20b4678d849 [term 1 LEADER]: Becoming Leader. State: Replica: 19f08364540043328e9da20b4678d849, State: Running, Role: LEADER
I20260929 18:35:02.755249 10729 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19f08364540043328e9da20b4678d849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19f08364540043328e9da20b4678d849" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 40053 } }
I20260929 18:35:02.755944 10724 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19f08364540043328e9da20b4678d849 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:35:02.757150 10730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19f08364540043328e9da20b4678d849 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19f08364540043328e9da20b4678d849" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19f08364540043328e9da20b4678d849" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 40053 } } }
I20260929 18:35:02.757336 10730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19f08364540043328e9da20b4678d849 [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:02.757150 10731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19f08364540043328e9da20b4678d849 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19f08364540043328e9da20b4678d849. Latest consensus state: current_term: 1 leader_uuid: "19f08364540043328e9da20b4678d849" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19f08364540043328e9da20b4678d849" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 40053 } } }
I20260929 18:35:02.757520 10731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19f08364540043328e9da20b4678d849 [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:02.758076 10737 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:35:02.761495 10737 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:35:02.768554 10737 catalog_manager.cc:1347] Generated new cluster ID: 377ac147c26146a19b833b376dce265d
I20260929 18:35:02.768654 10737 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:35:02.782089 10737 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:35:02.783349 10737 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:35:02.797526 10737 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19f08364540043328e9da20b4678d849: Generated new TSK 0
I20260929 18:35:02.798415 10737 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:35:02.809391 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 10656
2026-09-29T18:35:02Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1494 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:35:02 dist-test-slave-qcpv krb5kdc[10751](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:35:02 dist-test-slave-qcpv krb5kdc[10751](info): set up 2 sockets
Sep 29 18:35:02 dist-test-slave-qcpv krb5kdc[10751](info): commencing operation
krb5kdc: starting...
W20260929 18:35:03.895273 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:35:03 dist-test-slave-qcpv krb5kdc[10751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:35:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:35:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.207.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
I20260929 18:35:04.035074 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:43295
--webserver_interface=127.28.207.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:40155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:43295
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:04.154840 10764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:04.155359 10764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:04.155464 10764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:04.159720 10764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:35:04.159824 10764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:04.159861 10764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:04.159902 10764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:35:04.159943 10764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:35:04.165733 10764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:40155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:43295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:43295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:35:04.166981 10764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:04.168053 10764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:04.177104 10772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:04.177222 10770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:04.177233 10764 server_base.cc:1034] running on GCE node
W20260929 18:35:04.177398 10769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:04.178442 10764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:04.179737 10764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:04.180963 10764 hybrid_clock.cc:648] HybridClock initialized: now 1790706904180904 us; error 82 us; skew 500 ppm
Sep 29 18:35:04 dist-test-slave-qcpv krb5kdc[10751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706904, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:04.191859 10764 init.cc:377] Logged in from keytab as kudu/127.28.207.126@KRBTEST.COM (short username kudu)
I20260929 18:35:04.192899 10764 webserver.cc:460] Webserver started at http://127.28.207.126:41305/ using document root <none> and password file <none>
I20260929 18:35:04.193454 10764 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:04.193518 10764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:04.193687 10764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:04.195861 10764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "bf5d34d45bd44585ac555847b1786036"
format_stamp: "Formatted at 2026-09-29 18:35:04 on dist-test-slave-qcpv"
server_key: "c58ab8fc1bc9514d7182bb0d2c81a373"
server_key_iv: "c619fbd511652c0d466eef07620da584"
server_key_version: "encryptionkey@0"
I20260929 18:35:04.196395 10764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "bf5d34d45bd44585ac555847b1786036"
format_stamp: "Formatted at 2026-09-29 18:35:04 on dist-test-slave-qcpv"
server_key: "c58ab8fc1bc9514d7182bb0d2c81a373"
server_key_iv: "c619fbd511652c0d466eef07620da584"
server_key_version: "encryptionkey@0"
I20260929 18:35:04.200393 10764 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:35:04.203241 10779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:04.204829 10764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 18:35:04.204960 10764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "bf5d34d45bd44585ac555847b1786036"
format_stamp: "Formatted at 2026-09-29 18:35:04 on dist-test-slave-qcpv"
server_key: "c58ab8fc1bc9514d7182bb0d2c81a373"
server_key_iv: "c619fbd511652c0d466eef07620da584"
server_key_version: "encryptionkey@0"
I20260929 18:35:04.205066 10764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:04.218492 10764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:04.221488 10764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:04.221645 10764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:04.228979 10764 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:43295
I20260929 18:35:04.229413 10831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:43295 every 8 connection(s)
I20260929 18:35:04.230027 10764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:35:04.233331 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 10764
I20260929 18:35:04.233471 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:35:04.233678 10832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:04.233855 29501 external_mini_cluster.cc:1428] Setting key efa092d631e37b675ba8912706ab8959
I20260929 18:35:04.241165 10832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036: Bootstrap starting.
Sep 29 18:35:04 dist-test-slave-qcpv krb5kdc[10751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:35:04.243811 10832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:04.244747 10832 log.cc:824] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:04.247256 10832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036: No bootstrap required, opened a new log
I20260929 18:35:04.249487 10835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:04.235676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54606 (local address 127.28.207.126:43295)
0929 18:35:04.236254 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:04.236272 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:04.236314 (+ 42us) server_negotiation.cc:407] Connection header received
0929 18:35:04.237082 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:04.237100 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:04.237486 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:04.237880 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:04.238727 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:04.239827 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:04.240690 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:04.241182 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:04.241357 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:04.243934 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:04.243941 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:04.243960 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:04.243993 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:04.246338 (+ 2345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:04.246817 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:04.246822 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:04.246828 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:04.247006 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:04.247291 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:04.247295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:04.247297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:04.247863 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:04.248092 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:04.248359 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 18:35:04.248679 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":187,"threads_started":1}
I20260929 18:35:04.251654 10832 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5d34d45bd44585ac555847b1786036" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43295 } }
I20260929 18:35:04.251917 10832 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:04.252023 10832 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf5d34d45bd44585ac555847b1786036, State: Initialized, Role: FOLLOWER
I20260929 18:35:04.252517 10832 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5d34d45bd44585ac555847b1786036" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43295 } }
I20260929 18:35:04.252686 10832 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:35:04.252770 10832 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:35:04.252898 10832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:04.253880 10832 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5d34d45bd44585ac555847b1786036" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43295 } }
I20260929 18:35:04.254232 10832 leader_election.cc:302] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf5d34d45bd44585ac555847b1786036; no voters:
I20260929 18:35:04.254608 10832 leader_election.cc:288] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:35:04.254832 10837 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:04.255206 10837 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [term 1 LEADER]: Becoming Leader. State: Replica: bf5d34d45bd44585ac555847b1786036, State: Running, Role: LEADER
I20260929 18:35:04.255596 10837 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5d34d45bd44585ac555847b1786036" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43295 } }
I20260929 18:35:04.255923 10832 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:35:04.257499 10839 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf5d34d45bd44585ac555847b1786036. Latest consensus state: current_term: 1 leader_uuid: "bf5d34d45bd44585ac555847b1786036" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5d34d45bd44585ac555847b1786036" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43295 } } }
I20260929 18:35:04.257645 10839 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:04.257965 10838 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf5d34d45bd44585ac555847b1786036" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5d34d45bd44585ac555847b1786036" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43295 } } }
I20260929 18:35:04.258064 10838 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [sys.catalog]: This master's current role is: LEADER
W20260929 18:35:04.261436 10852 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:35:04.261533 10852 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:35:04.261617 10847 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:35:04.262462 10847 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:35:04.268316 10847 catalog_manager.cc:1347] Generated new cluster ID: 0994ef4f2f244e3b854dd0340735201e
I20260929 18:35:04.268411 10847 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:35:04.287906 10847 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:35:04.288951 10847 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:35:04.297763 10847 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036: Generated new TSK 0
I20260929 18:35:04.298655 10847 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.207.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab.
Entry for principal kudu/127.28.207.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab.
I20260929 18:35:04.342742 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.65:0
--local_ip_for_outbound_sockets=127.28.207.65
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:43295
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:40155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:04.454965 10858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:04.455253 10858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:04.455297 10858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:04.459630 10858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:04.459695 10858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:04.459754 10858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.65
I20260929 18:35:04.465029 10858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:40155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:43295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.65
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:35:04.466231 10858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:04.467324 10858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:04.477837 10864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:04.478018 10858 server_base.cc:1034] running on GCE node
W20260929 18:35:04.477865 10863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:04.478173 10866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:04.478693 10858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:04.479594 10858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:04.480801 10858 hybrid_clock.cc:648] HybridClock initialized: now 1790706904480766 us; error 53 us; skew 500 ppm
Sep 29 18:35:04 dist-test-slave-qcpv krb5kdc[10751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706904, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:04.484180 10858 init.cc:377] Logged in from keytab as kudu/127.28.207.65@KRBTEST.COM (short username kudu)
I20260929 18:35:04.485289 10858 webserver.cc:460] Webserver started at http://127.28.207.65:35325/ using document root <none> and password file <none>
I20260929 18:35:04.486014 10858 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:04.486110 10858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:04.486377 10858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:04.488850 10858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/data/instance:
uuid: "7762bdb5642b48ceb9b2e3b59c82d078"
format_stamp: "Formatted at 2026-09-29 18:35:04 on dist-test-slave-qcpv"
server_key: "92dc09a9b2a3ece0a264009f308d9b30"
server_key_iv: "3a1b1d59037ff84b88b22cd467354602"
server_key_version: "encryptionkey@0"
I20260929 18:35:04.489477 10858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance:
uuid: "7762bdb5642b48ceb9b2e3b59c82d078"
format_stamp: "Formatted at 2026-09-29 18:35:04 on dist-test-slave-qcpv"
server_key: "92dc09a9b2a3ece0a264009f308d9b30"
server_key_iv: "3a1b1d59037ff84b88b22cd467354602"
server_key_version: "encryptionkey@0"
I20260929 18:35:04.493882 10858 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260929 18:35:04.496965 10873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:04.498503 10858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:35:04.498677 10858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/wal
uuid: "7762bdb5642b48ceb9b2e3b59c82d078"
format_stamp: "Formatted at 2026-09-29 18:35:04 on dist-test-slave-qcpv"
server_key: "92dc09a9b2a3ece0a264009f308d9b30"
server_key_iv: "3a1b1d59037ff84b88b22cd467354602"
server_key_version: "encryptionkey@0"
I20260929 18:35:04.498805 10858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:04.514086 10858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:04.517719 10858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:04.517948 10858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:04.518625 10858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:35:04.519686 10858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:35:04.519757 10858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:04.519845 10858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:35:04.519889 10858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:04.530687 10858 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.65:40297
I20260929 18:35:04.530772 10986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.65:40297 every 8 connection(s)
I20260929 18:35:04.531790 10858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
I20260929 18:35:04.539857 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 10858
I20260929 18:35:04.540000 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance
I20260929 18:35:04.540354 29501 external_mini_cluster.cc:1428] Setting key b8f623839889c6ca884e2ab51aa7b11a
Sep 29 18:35:04 dist-test-slave-qcpv krb5kdc[10751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706904, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:35:04.549667 10835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:04.534235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:33973 (local address 127.28.207.126:43295)
0929 18:35:04.534487 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:04.534493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:04.536009 (+ 1516us) server_negotiation.cc:407] Connection header received
0929 18:35:04.537499 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:04.537504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:04.537579 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:04.537704 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:04.539387 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:04.540099 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:04.541602 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:04.541948 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:04.542067 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:04.545617 (+ 3550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:04.545622 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:04.545625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:04.545661 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:04.547401 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:04.547940 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:04.547944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:04.547945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:04.548031 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:04.548432 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:04.548435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:04.548437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:04.548655 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:04.548766 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:04.549295 (+ 529us) server_negotiation.cc:299] Negotiation successful
0929 18:35:04.549473 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260929 18:35:04.550477 10989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:04.534765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.65:33973)
0929 18:35:04.535801 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:04.535876 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:04.537193 (+ 1317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:04.537976 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:04.537985 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:04.538593 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:04.539213 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:04.539233 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:04.540255 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:04.540259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:04.541469 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:04.541481 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:04.542158 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:04.542164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:04.542418 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:04.543260 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:04.543287 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:04.545422 (+ 2135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:04.547526 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:04.547533 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:04.547549 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:04.547826 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:04.548133 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:04.548136 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:04.548138 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:04.548324 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:04.548764 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:04.548771 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:04.549041 (+ 270us) client_negotiation.cc:769] Sending connection context
0929 18:35:04.549239 (+ 198us) client_negotiation.cc:240] Negotiation successful
0929 18:35:04.549593 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"thread_start_us":181,"threads_started":1}
WARNING: no policy specified for kudu/127.28.207.66@KRBTEST.COM; defaulting to no policy
I20260929 18:35:04.552217 10987 heartbeater.cc:347] Connected to a master server at 127.28.207.126:43295
I20260929 18:35:04.552511 10987 heartbeater.cc:464] Registering TS with master...
I20260929 18:35:04.553095 10987 heartbeater.cc:511] Master 127.28.207.126:43295 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.66@KRBTEST.COM" created.
I20260929 18:35:04.555199 10796 ts_manager.cc:194] Registered new tserver with Master: 7762bdb5642b48ceb9b2e3b59c82d078 (127.28.207.65:40297)
I20260929 18:35:04.557024 10796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.65@KRBTEST.COM'} at 127.28.207.65:33973
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab.
Entry for principal kudu/127.28.207.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab.
I20260929 18:35:04.572383 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.66:0
--local_ip_for_outbound_sockets=127.28.207.66
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:43295
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:40155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:04.694377 10992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:04.694720 10992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:04.694809 10992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:04.699023 10992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:04.699102 10992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:04.699193 10992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.66
I20260929 18:35:04.704818 10992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:40155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:43295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.66
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:35:04.706200 10992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:04.707777 10992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:04.719039 11000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:04.719039 10998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:04.719084 10997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:04.719548 10992 server_base.cc:1034] running on GCE node
I20260929 18:35:04.720041 10992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:04.720741 10992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:04.721928 10992 hybrid_clock.cc:648] HybridClock initialized: now 1790706904721907 us; error 44 us; skew 500 ppm
Sep 29 18:35:04 dist-test-slave-qcpv krb5kdc[10751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706904, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:04.725219 10992 init.cc:377] Logged in from keytab as kudu/127.28.207.66@KRBTEST.COM (short username kudu)
I20260929 18:35:04.726318 10992 webserver.cc:460] Webserver started at http://127.28.207.66:35493/ using document root <none> and password file <none>
I20260929 18:35:04.727010 10992 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:04.727080 10992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:04.727278 10992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:04.729647 10992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/data/instance:
uuid: "a0da9e3be899425d8c1d8b3bab353ea9"
format_stamp: "Formatted at 2026-09-29 18:35:04 on dist-test-slave-qcpv"
server_key: "b1e05f7fc82c94fd4a24e39501dc47d6"
server_key_iv: "08d7667af4b3f8e91164defd6f9d55a4"
server_key_version: "encryptionkey@0"
I20260929 18:35:04.730211 10992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance:
uuid: "a0da9e3be899425d8c1d8b3bab353ea9"
format_stamp: "Formatted at 2026-09-29 18:35:04 on dist-test-slave-qcpv"
server_key: "b1e05f7fc82c94fd4a24e39501dc47d6"
server_key_iv: "08d7667af4b3f8e91164defd6f9d55a4"
server_key_version: "encryptionkey@0"
I20260929 18:35:04.736460 10992 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 18:35:04.740589 11007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:04.742324 10992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260929 18:35:04.742440 10992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/wal
uuid: "a0da9e3be899425d8c1d8b3bab353ea9"
format_stamp: "Formatted at 2026-09-29 18:35:04 on dist-test-slave-qcpv"
server_key: "b1e05f7fc82c94fd4a24e39501dc47d6"
server_key_iv: "08d7667af4b3f8e91164defd6f9d55a4"
server_key_version: "encryptionkey@0"
I20260929 18:35:04.742604 10992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:04.776093 10992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:04.779913 10992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:04.780160 10992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:04.780872 10992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:35:04.781839 10992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:35:04.781890 10992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:04.781934 10992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:35:04.781950 10992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:04.793533 10992 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.66:32883
I20260929 18:35:04.793558 11120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.66:32883 every 8 connection(s)
I20260929 18:35:04.794674 10992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
I20260929 18:35:04.800769 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 10992
I20260929 18:35:04.800967 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance
I20260929 18:35:04.801324 29501 external_mini_cluster.cc:1428] Setting key 9bca7555e206bed7600ec9bf2bf66dfc
Sep 29 18:35:04 dist-test-slave-qcpv krb5kdc[10751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706904, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:35:04.811178 10835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:04.797119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:56491 (local address 127.28.207.126:43295)
0929 18:35:04.797256 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:04.797267 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:04.798380 (+ 1113us) server_negotiation.cc:407] Connection header received
0929 18:35:04.800024 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:04.800027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:04.800089 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:04.800170 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:04.801603 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:04.802316 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:04.803769 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:04.804013 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:04.804128 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:04.807306 (+ 3178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:04.807310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:04.807315 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:04.807343 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:04.808950 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:04.809551 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:04.809564 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:04.809570 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:04.809643 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:04.809988 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:04.809991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:04.809993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:04.810184 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:04.810301 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:04.810850 (+ 549us) server_negotiation.cc:299] Negotiation successful
0929 18:35:04.811001 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:35:04.812012 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:04.797547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.66:56491)
0929 18:35:04.798194 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:04.798251 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:04.799783 (+ 1532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:04.800361 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:04.800369 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:04.800808 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:04.801458 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:04.801483 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:04.802472 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:04.802475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:04.803650 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:04.803660 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:04.804180 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:04.804184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:04.804353 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:04.805115 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:04.805132 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:04.807119 (+ 1987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:04.809092 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:04.809097 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:04.809113 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:04.809443 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:04.809736 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:04.809740 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:04.809742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:04.809887 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:04.810303 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:04.810310 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:04.810605 (+ 295us) client_negotiation.cc:769] Sending connection context
0929 18:35:04.810793 (+ 188us) client_negotiation.cc:240] Negotiation successful
0929 18:35:04.811158 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":158,"threads_started":1}
WARNING: no policy specified for kudu/127.28.207.67@KRBTEST.COM; defaulting to no policy
I20260929 18:35:04.813306 11121 heartbeater.cc:347] Connected to a master server at 127.28.207.126:43295
I20260929 18:35:04.813861 11121 heartbeater.cc:464] Registering TS with master...
I20260929 18:35:04.814522 11121 heartbeater.cc:511] Master 127.28.207.126:43295 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.67@KRBTEST.COM" created.
I20260929 18:35:04.816179 10796 ts_manager.cc:194] Registered new tserver with Master: a0da9e3be899425d8c1d8b3bab353ea9 (127.28.207.66:32883)
I20260929 18:35:04.817179 10796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.66@KRBTEST.COM'} at 127.28.207.66:56491
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab.
Entry for principal kudu/127.28.207.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab.
I20260929 18:35:04.834739 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.67:0
--local_ip_for_outbound_sockets=127.28.207.67
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:43295
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:40155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:04.957664 11126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:04.957952 11126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:04.957998 11126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:04.962211 11126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:04.962302 11126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:04.962394 11126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.67
I20260929 18:35:04.968046 11126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:40155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:43295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.67
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:35:04.969280 11126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:04.970394 11126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:04.981725 11134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:04.981813 11126 server_base.cc:1034] running on GCE node
W20260929 18:35:04.981824 11131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:04.981657 11132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:04.982605 11126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:04.983544 11126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:04.984833 11126 hybrid_clock.cc:648] HybridClock initialized: now 1790706904984794 us; error 57 us; skew 500 ppm
Sep 29 18:35:04 dist-test-slave-qcpv krb5kdc[10751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706904, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:04.988399 11126 init.cc:377] Logged in from keytab as kudu/127.28.207.67@KRBTEST.COM (short username kudu)
I20260929 18:35:04.989506 11126 webserver.cc:460] Webserver started at http://127.28.207.67:44107/ using document root <none> and password file <none>
I20260929 18:35:04.990207 11126 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:04.990299 11126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:04.990559 11126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:04.993180 11126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/data/instance:
uuid: "a572967618bb4a0eb37fab57e504e8de"
format_stamp: "Formatted at 2026-09-29 18:35:04 on dist-test-slave-qcpv"
server_key: "ce4429dddf07e0305486a764b5b1e942"
server_key_iv: "35a2dc5d245391a98481adc2decf0b7b"
server_key_version: "encryptionkey@0"
I20260929 18:35:04.993841 11126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance:
uuid: "a572967618bb4a0eb37fab57e504e8de"
format_stamp: "Formatted at 2026-09-29 18:35:04 on dist-test-slave-qcpv"
server_key: "ce4429dddf07e0305486a764b5b1e942"
server_key_iv: "35a2dc5d245391a98481adc2decf0b7b"
server_key_version: "encryptionkey@0"
I20260929 18:35:04.998464 11126 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 18:35:05.001835 11141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:05.003861 11126 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:35:05.004009 11126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/wal
uuid: "a572967618bb4a0eb37fab57e504e8de"
format_stamp: "Formatted at 2026-09-29 18:35:04 on dist-test-slave-qcpv"
server_key: "ce4429dddf07e0305486a764b5b1e942"
server_key_iv: "35a2dc5d245391a98481adc2decf0b7b"
server_key_version: "encryptionkey@0"
I20260929 18:35:05.004141 11126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:05.036324 11126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:05.040127 11126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:05.040371 11126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:05.041108 11126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:35:05.042086 11126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:35:05.042162 11126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:05.042235 11126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:35:05.042292 11126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:05.053880 11126 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.67:42373
I20260929 18:35:05.054004 11254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.67:42373 every 8 connection(s)
I20260929 18:35:05.054958 11126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
I20260929 18:35:05.063256 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 11126
I20260929 18:35:05.063367 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance
I20260929 18:35:05.063692 29501 external_mini_cluster.cc:1428] Setting key e46e03f7f52dca1a7eac8d4e9f9bc368
Sep 29 18:35:05 dist-test-slave-qcpv krb5kdc[10751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706904, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:35:05.071902 10835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:05.057366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:59417 (local address 127.28.207.126:43295)
0929 18:35:05.057545 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:05.057549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:05.058698 (+ 1149us) server_negotiation.cc:407] Connection header received
0929 18:35:05.060222 (+ 1524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:05.060225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:05.060285 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:05.060385 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:05.061868 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.062513 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.064108 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.064453 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.064600 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:05.068041 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:05.068044 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:05.068046 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:05.068076 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:05.069654 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:05.070218 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:05.070221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:05.070222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:05.070296 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:05.070674 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:05.070677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:05.070678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:05.070923 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:05.071021 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:05.071632 (+ 611us) server_negotiation.cc:299] Negotiation successful
0929 18:35:05.071776 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 18:35:05.072853 11257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:05.057891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.67:59417)
0929 18:35:05.058503 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:05.058553 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:05.059946 (+ 1393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:05.060622 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:05.060633 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:05.061083 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:05.061654 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:05.061668 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.062672 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.062675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:05.063937 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:05.063948 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.064673 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.064679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:05.064965 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:05.065818 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:05.065842 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:05.067846 (+ 2004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:05.069785 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:05.069789 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:05.069804 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:05.070097 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:05.070401 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:05.070403 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:05.070405 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:05.070568 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:05.071047 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:05.071052 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:05.071332 (+ 280us) client_negotiation.cc:769] Sending connection context
0929 18:35:05.071521 (+ 189us) client_negotiation.cc:240] Negotiation successful
0929 18:35:05.071849 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":158,"threads_started":1}
I20260929 18:35:05.074272 11255 heartbeater.cc:347] Connected to a master server at 127.28.207.126:43295
I20260929 18:35:05.074586 11255 heartbeater.cc:464] Registering TS with master...
I20260929 18:35:05.075314 11255 heartbeater.cc:511] Master 127.28.207.126:43295 requested a full tablet report, sending...
I20260929 18:35:05.076489 10796 ts_manager.cc:194] Registered new tserver with Master: a572967618bb4a0eb37fab57e504e8de (127.28.207.67:42373)
I20260929 18:35:05.077292 10796 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.67@KRBTEST.COM'} at 127.28.207.67:59417
I20260929 18:35:05.078945 29501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:35:05.089776 10835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:05.081080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54618 (local address 127.28.207.126:43295)
0929 18:35:05.081239 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:05.081243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:05.081452 (+ 209us) server_negotiation.cc:407] Connection header received
0929 18:35:05.081592 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:05.081595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:05.081640 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:05.081706 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:05.082514 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.083160 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.084115 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.084350 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.084437 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:05.085977 (+ 1540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:05.085980 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:05.085983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:05.086007 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:05.087823 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:05.088314 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:05.088318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:05.088319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:05.088435 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:05.088744 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:05.088746 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:05.088748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:05.089003 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:05.089160 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:05.089445 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 18:35:05.089594 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 18:35:05.093179 10795 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54618:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:35:05.096401 10795 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:35:05.113736 11269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:05.106680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:42373 (local address 127.0.0.1:41708)
0929 18:35:05.107638 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:05.107706 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:05.107995 (+ 289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:05.108998 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:05.109001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:05.109018 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:05.109153 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:05.109160 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.110908 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.110911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:05.112184 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:05.112194 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.113231 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.113234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:05.113330 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:05.113355 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 18:35:05.113439 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 18:35:05.113534 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"mutex_wait_us":95,"thread_start_us":102,"threads_started":1}
I20260929 18:35:05.113763 11267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:05.106882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41708 (local address 127.28.207.67:42373)
0929 18:35:05.108451 (+ 1569us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:05.108466 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:05.108487 (+ 21us) server_negotiation.cc:407] Connection header received
0929 18:35:05.108560 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:05.108564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:05.108774 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:05.109296 (+ 522us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:05.109472 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.110719 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.112330 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.113106 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.113204 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:05.113345 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:05.113489 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 18:35:05.113599 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1448,"thread_start_us":136,"threads_started":1}
I20260929 18:35:05.113937 11271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:05.107768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49204 (local address 127.28.207.66:32883)
0929 18:35:05.108386 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:05.108401 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:05.108426 (+ 25us) server_negotiation.cc:407] Connection header received
0929 18:35:05.108558 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:05.108564 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:05.108774 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:05.109002 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:05.109496 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.110703 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.112061 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.112819 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.112962 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:05.113126 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:05.113629 (+ 503us) server_negotiation.cc:299] Negotiation successful
0929 18:35:05.113738 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":182,"threads_started":1}
I20260929 18:35:05.114199 11266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:05.107135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:32883 (local address 127.0.0.1:49204)
0929 18:35:05.108310 (+ 1175us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:05.108325 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:05.108403 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:05.109160 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:05.109163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:05.109178 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:05.109298 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:05.109303 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.110883 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.110890 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:05.111890 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:05.111899 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.113182 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.113186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:05.113273 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:05.113298 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 18:35:05.113377 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 18:35:05.113476 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1054,"mutex_wait_us":36,"thread_start_us":116,"threads_started":1}
I20260929 18:35:05.114517 11268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:05.106960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:40297 (local address 127.0.0.1:55660)
0929 18:35:05.108026 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:05.108042 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:05.108147 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:05.108742 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:05.108749 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:05.108772 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:05.108925 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:05.108932 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.110244 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.110248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:05.111709 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:05.111720 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.112924 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.112929 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:05.113036 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:05.113074 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 18:35:05.113190 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 18:35:05.114362 (+ 1172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":989,"thread_start_us":103,"threads_started":1}
I20260929 18:35:05.114563 11270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:05.107638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55660 (local address 127.28.207.65:40297)
0929 18:35:05.108192 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:05.108200 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:05.108224 (+ 24us) server_negotiation.cc:407] Connection header received
0929 18:35:05.108305 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:05.108310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:05.108586 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:05.108760 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:05.109088 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.110105 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.111891 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.112765 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.112904 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:05.113091 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:05.114225 (+ 1134us) server_negotiation.cc:299] Negotiation successful
0929 18:35:05.114398 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":159,"threads_started":1}
I20260929 18:35:05.117821 10921 tablet_service.cc:1467] Processing CreateTablet for tablet de4df53be70d4f61af6ab797ffc6289c (DEFAULT_TABLE table=test-table [id=810d4080827f45e58f438201e946e151]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:05.117821 11055 tablet_service.cc:1467] Processing CreateTablet for tablet de4df53be70d4f61af6ab797ffc6289c (DEFAULT_TABLE table=test-table [id=810d4080827f45e58f438201e946e151]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:05.117775 11189 tablet_service.cc:1467] Processing CreateTablet for tablet de4df53be70d4f61af6ab797ffc6289c (DEFAULT_TABLE table=test-table [id=810d4080827f45e58f438201e946e151]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:05.119027 10921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de4df53be70d4f61af6ab797ffc6289c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:05.119027 11055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de4df53be70d4f61af6ab797ffc6289c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:05.119145 11189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de4df53be70d4f61af6ab797ffc6289c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:05.126181 11272 tablet_bootstrap.cc:492] T de4df53be70d4f61af6ab797ffc6289c P a572967618bb4a0eb37fab57e504e8de: Bootstrap starting.
I20260929 18:35:05.127802 11273 tablet_bootstrap.cc:492] T de4df53be70d4f61af6ab797ffc6289c P 7762bdb5642b48ceb9b2e3b59c82d078: Bootstrap starting.
I20260929 18:35:05.127789 11274 tablet_bootstrap.cc:492] T de4df53be70d4f61af6ab797ffc6289c P a0da9e3be899425d8c1d8b3bab353ea9: Bootstrap starting.
I20260929 18:35:05.128957 11272 tablet_bootstrap.cc:654] T de4df53be70d4f61af6ab797ffc6289c P a572967618bb4a0eb37fab57e504e8de: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:05.130062 11273 tablet_bootstrap.cc:654] T de4df53be70d4f61af6ab797ffc6289c P 7762bdb5642b48ceb9b2e3b59c82d078: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:05.130491 11274 tablet_bootstrap.cc:654] T de4df53be70d4f61af6ab797ffc6289c P a0da9e3be899425d8c1d8b3bab353ea9: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:05.130522 11272 log.cc:824] T de4df53be70d4f61af6ab797ffc6289c P a572967618bb4a0eb37fab57e504e8de: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:05.130946 11273 log.cc:824] T de4df53be70d4f61af6ab797ffc6289c P 7762bdb5642b48ceb9b2e3b59c82d078: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:05.131323 11274 log.cc:824] T de4df53be70d4f61af6ab797ffc6289c P a0da9e3be899425d8c1d8b3bab353ea9: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:05.133824 11272 tablet_bootstrap.cc:492] T de4df53be70d4f61af6ab797ffc6289c P a572967618bb4a0eb37fab57e504e8de: No bootstrap required, opened a new log
I20260929 18:35:05.133828 11274 tablet_bootstrap.cc:492] T de4df53be70d4f61af6ab797ffc6289c P a0da9e3be899425d8c1d8b3bab353ea9: No bootstrap required, opened a new log
I20260929 18:35:05.134045 11274 ts_tablet_manager.cc:1397] T de4df53be70d4f61af6ab797ffc6289c P a0da9e3be899425d8c1d8b3bab353ea9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 18:35:05.134045 11272 ts_tablet_manager.cc:1397] T de4df53be70d4f61af6ab797ffc6289c P a572967618bb4a0eb37fab57e504e8de: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260929 18:35:05.136354 11273 tablet_bootstrap.cc:492] T de4df53be70d4f61af6ab797ffc6289c P 7762bdb5642b48ceb9b2e3b59c82d078: No bootstrap required, opened a new log
I20260929 18:35:05.136579 11273 ts_tablet_manager.cc:1397] T de4df53be70d4f61af6ab797ffc6289c P 7762bdb5642b48ceb9b2e3b59c82d078: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260929 18:35:05.138466 11272 raft_consensus.cc:348] T de4df53be70d4f61af6ab797ffc6289c P a572967618bb4a0eb37fab57e504e8de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572967618bb4a0eb37fab57e504e8de" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 42373 } } peers { permanent_uuid: "7762bdb5642b48ceb9b2e3b59c82d078" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 40297 } } peers { permanent_uuid: "a0da9e3be899425d8c1d8b3bab353ea9" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 32883 } }
I20260929 18:35:05.138466 11274 raft_consensus.cc:348] T de4df53be70d4f61af6ab797ffc6289c P a0da9e3be899425d8c1d8b3bab353ea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572967618bb4a0eb37fab57e504e8de" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 42373 } } peers { permanent_uuid: "7762bdb5642b48ceb9b2e3b59c82d078" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 40297 } } peers { permanent_uuid: "a0da9e3be899425d8c1d8b3bab353ea9" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 32883 } }
I20260929 18:35:05.138787 11272 raft_consensus.cc:374] T de4df53be70d4f61af6ab797ffc6289c P a572967618bb4a0eb37fab57e504e8de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:05.138787 11274 raft_consensus.cc:374] T de4df53be70d4f61af6ab797ffc6289c P a0da9e3be899425d8c1d8b3bab353ea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:05.138849 11274 raft_consensus.cc:729] T de4df53be70d4f61af6ab797ffc6289c P a0da9e3be899425d8c1d8b3bab353ea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0da9e3be899425d8c1d8b3bab353ea9, State: Initialized, Role: FOLLOWER
I20260929 18:35:05.138849 11272 raft_consensus.cc:729] T de4df53be70d4f61af6ab797ffc6289c P a572967618bb4a0eb37fab57e504e8de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a572967618bb4a0eb37fab57e504e8de, State: Initialized, Role: FOLLOWER
I20260929 18:35:05.139411 11272 consensus_queue.cc:260] T de4df53be70d4f61af6ab797ffc6289c P a572967618bb4a0eb37fab57e504e8de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572967618bb4a0eb37fab57e504e8de" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 42373 } } peers { permanent_uuid: "7762bdb5642b48ceb9b2e3b59c82d078" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 40297 } } peers { permanent_uuid: "a0da9e3be899425d8c1d8b3bab353ea9" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 32883 } }
I20260929 18:35:05.139411 11274 consensus_queue.cc:260] T de4df53be70d4f61af6ab797ffc6289c P a0da9e3be899425d8c1d8b3bab353ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572967618bb4a0eb37fab57e504e8de" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 42373 } } peers { permanent_uuid: "7762bdb5642b48ceb9b2e3b59c82d078" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 40297 } } peers { permanent_uuid: "a0da9e3be899425d8c1d8b3bab353ea9" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 32883 } }
I20260929 18:35:05.140515 11121 heartbeater.cc:503] Master 127.28.207.126:43295 was elected leader, sending a full tablet report...
I20260929 18:35:05.140600 11272 ts_tablet_manager.cc:1428] T de4df53be70d4f61af6ab797ffc6289c P a572967618bb4a0eb37fab57e504e8de: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20260929 18:35:05.140600 11274 ts_tablet_manager.cc:1428] T de4df53be70d4f61af6ab797ffc6289c P a0da9e3be899425d8c1d8b3bab353ea9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 18:35:05.141031 11255 heartbeater.cc:503] Master 127.28.207.126:43295 was elected leader, sending a full tablet report...
I20260929 18:35:05.141067 11273 raft_consensus.cc:348] T de4df53be70d4f61af6ab797ffc6289c P 7762bdb5642b48ceb9b2e3b59c82d078 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572967618bb4a0eb37fab57e504e8de" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 42373 } } peers { permanent_uuid: "7762bdb5642b48ceb9b2e3b59c82d078" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 40297 } } peers { permanent_uuid: "a0da9e3be899425d8c1d8b3bab353ea9" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 32883 } }
I20260929 18:35:05.141389 11273 raft_consensus.cc:374] T de4df53be70d4f61af6ab797ffc6289c P 7762bdb5642b48ceb9b2e3b59c82d078 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:05.141459 11273 raft_consensus.cc:729] T de4df53be70d4f61af6ab797ffc6289c P 7762bdb5642b48ceb9b2e3b59c82d078 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7762bdb5642b48ceb9b2e3b59c82d078, State: Initialized, Role: FOLLOWER
I20260929 18:35:05.141959 11273 consensus_queue.cc:260] T de4df53be70d4f61af6ab797ffc6289c P 7762bdb5642b48ceb9b2e3b59c82d078 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572967618bb4a0eb37fab57e504e8de" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 42373 } } peers { permanent_uuid: "7762bdb5642b48ceb9b2e3b59c82d078" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 40297 } } peers { permanent_uuid: "a0da9e3be899425d8c1d8b3bab353ea9" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 32883 } }
I20260929 18:35:05.143046 11273 ts_tablet_manager.cc:1428] T de4df53be70d4f61af6ab797ffc6289c P 7762bdb5642b48ceb9b2e3b59c82d078: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260929 18:35:05.143384 10987 heartbeater.cc:503] Master 127.28.207.126:43295 was elected leader, sending a full tablet report...
W20260929 18:35:05.283715 10988 tablet.cc:2369] T de4df53be70d4f61af6ab797ffc6289c P 7762bdb5642b48ceb9b2e3b59c82d078: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:35:05.296394 11122 tablet.cc:2369] T de4df53be70d4f61af6ab797ffc6289c P a0da9e3be899425d8c1d8b3bab353ea9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:35:05.306044 11256 tablet.cc:2369] T de4df53be70d4f61af6ab797ffc6289c P a572967618bb4a0eb37fab57e504e8de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:35:05.320307 11279 raft_consensus.cc:482] T de4df53be70d4f61af6ab797ffc6289c P a0da9e3be899425d8c1d8b3bab353ea9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:35:05.320608 11279 raft_consensus.cc:504] T de4df53be70d4f61af6ab797ffc6289c P a0da9e3be899425d8c1d8b3bab353ea9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572967618bb4a0eb37fab57e504e8de" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 42373 } } peers { permanent_uuid: "7762bdb5642b48ceb9b2e3b59c82d078" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 40297 } } peers { permanent_uuid: "a0da9e3be899425d8c1d8b3bab353ea9" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 32883 } }
I20260929 18:35:05.321882 11279 leader_election.cc:288] T de4df53be70d4f61af6ab797ffc6289c P a0da9e3be899425d8c1d8b3bab353ea9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a572967618bb4a0eb37fab57e504e8de (127.28.207.67:42373), 7762bdb5642b48ceb9b2e3b59c82d078 (127.28.207.65:40297)
I20260929 18:35:05.326407 11270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:05.322394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:48827 (local address 127.28.207.65:40297)
0929 18:35:05.322525 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:05.322531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:05.322795 (+ 264us) server_negotiation.cc:407] Connection header received
0929 18:35:05.322978 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:05.322981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:05.323025 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:05.323133 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:05.323544 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.324051 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.325263 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.325684 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.325843 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:05.325914 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:05.326180 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 18:35:05.326268 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 18:35:05.326417 11267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:05.322330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:56339 (local address 127.28.207.67:42373)
0929 18:35:05.322495 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:05.322500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:05.322670 (+ 170us) server_negotiation.cc:407] Connection header received
0929 18:35:05.322943 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:05.322949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:05.323013 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:05.323076 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:05.323472 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.324055 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.325284 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.325691 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.325927 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:05.325999 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:05.326215 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 18:35:05.326294 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 18:35:05.326467 11281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:05.322125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:42373 (local address 127.28.207.66:56339)
0929 18:35:05.322525 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:05.322542 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:05.322728 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:05.323130 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:05.323133 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:05.323153 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:05.323351 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:05.323356 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.324191 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.324194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:05.325125 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:05.325134 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.325932 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.325939 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:05.326047 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:05.326080 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 18:35:05.326183 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 18:35:05.326274 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":10,"spinlock_wait_cycles":22272,"thread_start_us":104,"threads_started":1}
I20260929 18:35:05.326437 11282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:05.322266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:40297 (local address 127.28.207.66:48827)
0929 18:35:05.322681 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:05.322694 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:05.322853 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:05.323163 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:05.323166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:05.323223 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:05.323397 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:05.323403 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.324172 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.324177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:05.325112 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:05.325121 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.325913 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.325917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:05.326024 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:05.326060 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 18:35:05.326124 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 18:35:05.326228 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":26,"spinlock_wait_cycles":5632,"thread_start_us":97,"threads_started":1}
I20260929 18:35:05.327296 11209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de4df53be70d4f61af6ab797ffc6289c" candidate_uuid: "a0da9e3be899425d8c1d8b3bab353ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a572967618bb4a0eb37fab57e504e8de" is_pre_election: true
I20260929 18:35:05.327428 10941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de4df53be70d4f61af6ab797ffc6289c" candidate_uuid: "a0da9e3be899425d8c1d8b3bab353ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7762bdb5642b48ceb9b2e3b59c82d078" is_pre_election: true
I20260929 18:35:05.327731 10941 raft_consensus.cc:2454] T de4df53be70d4f61af6ab797ffc6289c P 7762bdb5642b48ceb9b2e3b59c82d078 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0da9e3be899425d8c1d8b3bab353ea9 in term 0.
I20260929 18:35:05.327731 11209 raft_consensus.cc:2454] T de4df53be70d4f61af6ab797ffc6289c P a572967618bb4a0eb37fab57e504e8de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0da9e3be899425d8c1d8b3bab353ea9 in term 0.
I20260929 18:35:05.328406 11011 leader_election.cc:302] T de4df53be70d4f61af6ab797ffc6289c P a0da9e3be899425d8c1d8b3bab353ea9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7762bdb5642b48ceb9b2e3b59c82d078, a0da9e3be899425d8c1d8b3bab353ea9, a572967618bb4a0eb37fab57e504e8de; no voters:
I20260929 18:35:05.328816 11279 raft_consensus.cc:2781] T de4df53be70d4f61af6ab797ffc6289c P a0da9e3be899425d8c1d8b3bab353ea9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:35:05.328958 11279 raft_consensus.cc:482] T de4df53be70d4f61af6ab797ffc6289c P a0da9e3be899425d8c1d8b3bab353ea9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:35:05.329053 11279 raft_consensus.cc:3037] T de4df53be70d4f61af6ab797ffc6289c P a0da9e3be899425d8c1d8b3bab353ea9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:05.331468 11279 raft_consensus.cc:504] T de4df53be70d4f61af6ab797ffc6289c P a0da9e3be899425d8c1d8b3bab353ea9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572967618bb4a0eb37fab57e504e8de" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 42373 } } peers { permanent_uuid: "7762bdb5642b48ceb9b2e3b59c82d078" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 40297 } } peers { permanent_uuid: "a0da9e3be899425d8c1d8b3bab353ea9" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 32883 } }
I20260929 18:35:05.331939 11279 leader_election.cc:288] T de4df53be70d4f61af6ab797ffc6289c P a0da9e3be899425d8c1d8b3bab353ea9 [CANDIDATE]: Term 1 election: Requested vote from peers a572967618bb4a0eb37fab57e504e8de (127.28.207.67:42373), 7762bdb5642b48ceb9b2e3b59c82d078 (127.28.207.65:40297)
I20260929 18:35:05.332651 11209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de4df53be70d4f61af6ab797ffc6289c" candidate_uuid: "a0da9e3be899425d8c1d8b3bab353ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a572967618bb4a0eb37fab57e504e8de"
I20260929 18:35:05.332652 10941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de4df53be70d4f61af6ab797ffc6289c" candidate_uuid: "a0da9e3be899425d8c1d8b3bab353ea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7762bdb5642b48ceb9b2e3b59c82d078"
I20260929 18:35:05.332957 10941 raft_consensus.cc:3037] T de4df53be70d4f61af6ab797ffc6289c P 7762bdb5642b48ceb9b2e3b59c82d078 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:05.332957 11209 raft_consensus.cc:3037] T de4df53be70d4f61af6ab797ffc6289c P a572967618bb4a0eb37fab57e504e8de [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:05.334836 10941 raft_consensus.cc:2454] T de4df53be70d4f61af6ab797ffc6289c P 7762bdb5642b48ceb9b2e3b59c82d078 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0da9e3be899425d8c1d8b3bab353ea9 in term 1.
I20260929 18:35:05.334830 11209 raft_consensus.cc:2454] T de4df53be70d4f61af6ab797ffc6289c P a572967618bb4a0eb37fab57e504e8de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0da9e3be899425d8c1d8b3bab353ea9 in term 1.
I20260929 18:35:05.335256 11010 leader_election.cc:302] T de4df53be70d4f61af6ab797ffc6289c P a0da9e3be899425d8c1d8b3bab353ea9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0da9e3be899425d8c1d8b3bab353ea9, a572967618bb4a0eb37fab57e504e8de; no voters:
I20260929 18:35:05.335507 11279 raft_consensus.cc:2781] T de4df53be70d4f61af6ab797ffc6289c P a0da9e3be899425d8c1d8b3bab353ea9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:05.335819 11279 raft_consensus.cc:686] T de4df53be70d4f61af6ab797ffc6289c P a0da9e3be899425d8c1d8b3bab353ea9 [term 1 LEADER]: Becoming Leader. State: Replica: a0da9e3be899425d8c1d8b3bab353ea9, State: Running, Role: LEADER
I20260929 18:35:05.336275 11279 consensus_queue.cc:237] T de4df53be70d4f61af6ab797ffc6289c P a0da9e3be899425d8c1d8b3bab353ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572967618bb4a0eb37fab57e504e8de" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 42373 } } peers { permanent_uuid: "7762bdb5642b48ceb9b2e3b59c82d078" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 40297 } } peers { permanent_uuid: "a0da9e3be899425d8c1d8b3bab353ea9" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 32883 } }
I20260929 18:35:05.339850 10795 catalog_manager.cc:5495] T de4df53be70d4f61af6ab797ffc6289c P a0da9e3be899425d8c1d8b3bab353ea9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0da9e3be899425d8c1d8b3bab353ea9 (127.28.207.66). New cstate: current_term: 1 leader_uuid: "a0da9e3be899425d8c1d8b3bab353ea9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572967618bb4a0eb37fab57e504e8de" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 42373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7762bdb5642b48ceb9b2e3b59c82d078" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 40297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0da9e3be899425d8c1d8b3bab353ea9" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 32883 } health_report { overall_health: HEALTHY } } }
I20260929 18:35:05.353216 11271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:05.349114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41566 (local address 127.28.207.66:32883)
0929 18:35:05.349274 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:05.349279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:05.349297 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:35:05.349467 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:05.349470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:05.349537 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:05.349655 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:35:05.349918 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.350646 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.351614 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.351836 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.351960 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:05.352280 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:35:05.352809 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:35:05.352955 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:05.352997 (+ 42us) server_negotiation.cc:299] Negotiation successful
0929 18:35:05.353054 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 18:35:05.360721 11209 raft_consensus.cc:1261] T de4df53be70d4f61af6ab797ffc6289c P a572967618bb4a0eb37fab57e504e8de [term 1 FOLLOWER]: Refusing update from remote peer a0da9e3be899425d8c1d8b3bab353ea9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:35:05.360798 10941 raft_consensus.cc:1261] T de4df53be70d4f61af6ab797ffc6289c P 7762bdb5642b48ceb9b2e3b59c82d078 [term 1 FOLLOWER]: Refusing update from remote peer a0da9e3be899425d8c1d8b3bab353ea9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:35:05.361641 11283 consensus_queue.cc:1035] T de4df53be70d4f61af6ab797ffc6289c P a0da9e3be899425d8c1d8b3bab353ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7762bdb5642b48ceb9b2e3b59c82d078" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:35:05.361945 11279 consensus_queue.cc:1035] T de4df53be70d4f61af6ab797ffc6289c P a0da9e3be899425d8c1d8b3bab353ea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a572967618bb4a0eb37fab57e504e8de" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 42373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:35:05.370393 11286 mvcc.cc:205] Tried to move back new op lower bound from 7334735484348325888 to 7334735484260057088. Current Snapshot: MvccSnapshot[applied={T|T < 7334735484348325888}]
I20260929 18:35:05.381656 10796 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:35:05.381858 10796 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:35:05.384507 10796 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036: Sending DeleteTablet for 3 replicas of tablet de4df53be70d4f61af6ab797ffc6289c
I20260929 18:35:05.385433 10921 tablet_service.cc:1514] Processing DeleteTablet for tablet de4df53be70d4f61af6ab797ffc6289c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:05 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:55660
I20260929 18:35:05.385433 11055 tablet_service.cc:1514] Processing DeleteTablet for tablet de4df53be70d4f61af6ab797ffc6289c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:05 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:49204
I20260929 18:35:05.385471 11189 tablet_service.cc:1514] Processing DeleteTablet for tablet de4df53be70d4f61af6ab797ffc6289c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:05 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:41708
I20260929 18:35:05.386013 11296 tablet_replica.cc:330] T de4df53be70d4f61af6ab797ffc6289c P a0da9e3be899425d8c1d8b3bab353ea9: stopping tablet replica
I20260929 18:35:05.386391 11296 raft_consensus.cc:2229] T de4df53be70d4f61af6ab797ffc6289c P a0da9e3be899425d8c1d8b3bab353ea9 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:35:05.386515 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 10858
I20260929 18:35:05.386687 11297 tablet_replica.cc:330] T de4df53be70d4f61af6ab797ffc6289c P 7762bdb5642b48ceb9b2e3b59c82d078: stopping tablet replica
I20260929 18:35:05.387218 11298 tablet_replica.cc:330] T de4df53be70d4f61af6ab797ffc6289c P a572967618bb4a0eb37fab57e504e8de: stopping tablet replica
I20260929 18:35:05.387570 11298 raft_consensus.cc:2229] T de4df53be70d4f61af6ab797ffc6289c P a572967618bb4a0eb37fab57e504e8de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:35:05.387879 11296 raft_consensus.cc:2258] T de4df53be70d4f61af6ab797ffc6289c P a0da9e3be899425d8c1d8b3bab353ea9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:35:05.388062 11298 raft_consensus.cc:2258] T de4df53be70d4f61af6ab797ffc6289c P a572967618bb4a0eb37fab57e504e8de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:35:05.389739 11296 ts_tablet_manager.cc:1905] T de4df53be70d4f61af6ab797ffc6289c P a0da9e3be899425d8c1d8b3bab353ea9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:35:05.392354 11296 ts_tablet_manager.cc:1918] T de4df53be70d4f61af6ab797ffc6289c P a0da9e3be899425d8c1d8b3bab353ea9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:35:05.392482 11296 log.cc:1196] T de4df53be70d4f61af6ab797ffc6289c P a0da9e3be899425d8c1d8b3bab353ea9: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/wal/wals/de4df53be70d4f61af6ab797ffc6289c
I20260929 18:35:05.392832 11296 ts_tablet_manager.cc:1939] T de4df53be70d4f61af6ab797ffc6289c P a0da9e3be899425d8c1d8b3bab353ea9: Deleting consensus metadata
I20260929 18:35:05.394237 11298 ts_tablet_manager.cc:1905] T de4df53be70d4f61af6ab797ffc6289c P a572967618bb4a0eb37fab57e504e8de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:35:05.394234 10780 catalog_manager.cc:4841] TS a0da9e3be899425d8c1d8b3bab353ea9 (127.28.207.66:32883): tablet de4df53be70d4f61af6ab797ffc6289c (table test-table [id=810d4080827f45e58f438201e946e151]) successfully deleted
W20260929 18:35:05.396308 10783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.207.65:40297 (error 108)
I20260929 18:35:05.396610 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 10992
I20260929 18:35:05.398311 11269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:05.397865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:40297 (local address 127.0.0.1:57412)
0929 18:35:05.398056 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:05.398181 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.65:40297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260929 18:35:05.398515 10783 catalog_manager.cc:4584] TS 7762bdb5642b48ceb9b2e3b59c82d078 (127.28.207.65:40297): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet de4df53be70d4f61af6ab797ffc6289c: Network error: Client connection negotiation failed: client connection to 127.28.207.65:40297: connect: Connection refused (error 111)
I20260929 18:35:05.398582 11298 ts_tablet_manager.cc:1918] T de4df53be70d4f61af6ab797ffc6289c P a572967618bb4a0eb37fab57e504e8de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:35:05.398782 11298 log.cc:1196] T de4df53be70d4f61af6ab797ffc6289c P a572967618bb4a0eb37fab57e504e8de: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/wal/wals/de4df53be70d4f61af6ab797ffc6289c
I20260929 18:35:05.399185 11298 ts_tablet_manager.cc:1939] T de4df53be70d4f61af6ab797ffc6289c P a572967618bb4a0eb37fab57e504e8de: Deleting consensus metadata
I20260929 18:35:05.400542 10782 catalog_manager.cc:4841] TS a572967618bb4a0eb37fab57e504e8de (127.28.207.67:42373): tablet de4df53be70d4f61af6ab797ffc6289c (table test-table [id=810d4080827f45e58f438201e946e151]) successfully deleted
I20260929 18:35:05.404145 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 11126
I20260929 18:35:05.411446 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 10764
I20260929 18:35:05.419649 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:43295
--webserver_interface=127.28.207.126
--webserver_port=41305
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:40155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:43295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:05.533666 11300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:05.534003 11300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:05.534087 11300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:05.539963 11300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:35:05.540072 11300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:05.540102 11300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:05.540136 11300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:35:05.540175 11300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:35:05.546648 11300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:40155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:43295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:43295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=41305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:35:05.547845 11300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:05.548846 11300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:05.563841 11309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:05.563926 11307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:05.563841 11306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:05.564327 11300 server_base.cc:1034] running on GCE node
I20260929 18:35:05.565194 11300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:05.566573 11300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:05.567772 11300 hybrid_clock.cc:648] HybridClock initialized: now 1790706905567726 us; error 70 us; skew 500 ppm
Sep 29 18:35:05 dist-test-slave-qcpv krb5kdc[10751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706905, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:05.571096 11300 init.cc:377] Logged in from keytab as kudu/127.28.207.126@KRBTEST.COM (short username kudu)
I20260929 18:35:05.572309 11300 webserver.cc:460] Webserver started at http://127.28.207.126:41305/ using document root <none> and password file <none>
I20260929 18:35:05.572994 11300 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:05.573077 11300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:05.577805 11300 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 18:35:05.580703 11316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:05.582320 11300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 18:35:05.582507 11300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "bf5d34d45bd44585ac555847b1786036"
format_stamp: "Formatted at 2026-09-29 18:35:04 on dist-test-slave-qcpv"
server_key: "c58ab8fc1bc9514d7182bb0d2c81a373"
server_key_iv: "c619fbd511652c0d466eef07620da584"
server_key_version: "encryptionkey@0"
I20260929 18:35:05.583292 11300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:05.606809 11300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:05.610848 11300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:05.611151 11300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:05.620147 11300 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:43295
I20260929 18:35:05.620478 11368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:43295 every 8 connection(s)
I20260929 18:35:05.622002 11300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:35:05.627254 11369 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:35:05.627894 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 11300
I20260929 18:35:05.628443 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.65:40297
--local_ip_for_outbound_sockets=127.28.207.65
--tserver_master_addrs=127.28.207.126:43295
--webserver_port=35325
--webserver_interface=127.28.207.65
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:40155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:35:05.629297 11369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036: Bootstrap starting.
I20260929 18:35:05.648149 11369 log.cc:824] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:05.657418 11369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:35:05.657858 11369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036: Bootstrap complete.
I20260929 18:35:05.662338 11369 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5d34d45bd44585ac555847b1786036" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43295 } }
I20260929 18:35:05.662817 11369 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf5d34d45bd44585ac555847b1786036, State: Initialized, Role: FOLLOWER
I20260929 18:35:05.663582 11369 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5d34d45bd44585ac555847b1786036" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43295 } }
I20260929 18:35:05.663784 11369 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:35:05.663878 11369 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:35:05.664026 11369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [term 1 FOLLOWER]: Advancing to term 2
I20260929 18:35:05.666489 11369 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5d34d45bd44585ac555847b1786036" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43295 } }
I20260929 18:35:05.666971 11369 leader_election.cc:302] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf5d34d45bd44585ac555847b1786036; no voters:
I20260929 18:35:05.667389 11369 leader_election.cc:288] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 18:35:05.667611 11373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 18:35:05.668000 11373 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [term 2 LEADER]: Becoming Leader. State: Replica: bf5d34d45bd44585ac555847b1786036, State: Running, Role: LEADER
I20260929 18:35:05.668422 11373 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5d34d45bd44585ac555847b1786036" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43295 } }
I20260929 18:35:05.669009 11369 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:35:05.670173 11374 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "bf5d34d45bd44585ac555847b1786036" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5d34d45bd44585ac555847b1786036" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43295 } } }
I20260929 18:35:05.670339 11374 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:05.670642 11375 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf5d34d45bd44585ac555847b1786036. Latest consensus state: current_term: 2 leader_uuid: "bf5d34d45bd44585ac555847b1786036" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5d34d45bd44585ac555847b1786036" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43295 } } }
I20260929 18:35:05.670744 11375 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:05.671629 11382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:35:05.676872 11382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:35:05.677621 11382 catalog_manager.cc:1259] Loaded cluster ID: 0994ef4f2f244e3b854dd0340735201e
I20260929 18:35:05.677695 11382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:35:05.679224 11382 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 18:35:05.684160 11382 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 18:35:05.691155 11382 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:35:05.692137 11382 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036: Loaded TSK: 0
I20260929 18:35:05.693528 11382 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 18:35:05.760787 11371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:05.761129 11371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:05.761211 11371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:05.765529 11371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:05.765627 11371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:05.765722 11371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.65
I20260929 18:35:05.771484 11371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:40155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.65:40297
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.207.65
--webserver_port=35325
--tserver_master_addrs=127.28.207.126:43295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.65
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:35:05.772903 11371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:05.774420 11371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:05.786661 11396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:05.786617 11399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:05.787214 11371 server_base.cc:1034] running on GCE node
W20260929 18:35:05.787215 11397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:05.787998 11371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:05.789027 11371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:05.790289 11371 hybrid_clock.cc:648] HybridClock initialized: now 1790706905790257 us; error 45 us; skew 500 ppm
Sep 29 18:35:05 dist-test-slave-qcpv krb5kdc[10751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706905, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:05.794346 11371 init.cc:377] Logged in from keytab as kudu/127.28.207.65@KRBTEST.COM (short username kudu)
I20260929 18:35:05.795738 11371 webserver.cc:460] Webserver started at http://127.28.207.65:35325/ using document root <none> and password file <none>
I20260929 18:35:05.796643 11371 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:05.796761 11371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:05.802233 11371 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:35:05.805337 11406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:05.807238 11371 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:35:05.807420 11371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/wal
uuid: "7762bdb5642b48ceb9b2e3b59c82d078"
format_stamp: "Formatted at 2026-09-29 18:35:04 on dist-test-slave-qcpv"
server_key: "92dc09a9b2a3ece0a264009f308d9b30"
server_key_iv: "3a1b1d59037ff84b88b22cd467354602"
server_key_version: "encryptionkey@0"
I20260929 18:35:05.808144 11371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:05.823885 11371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:05.828091 11371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:05.828343 11371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:05.829100 11371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:35:05.830513 11413 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 18:35:05.835541 11371 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 18:35:05.835685 11371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260929 18:35:05.835798 11371 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 18:35:05.838202 11371 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 18:35:05.838291 11371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260929 18:35:05.838719 11413 tablet_bootstrap.cc:492] T de4df53be70d4f61af6ab797ffc6289c P 7762bdb5642b48ceb9b2e3b59c82d078: Bootstrap starting.
I20260929 18:35:05.851624 11371 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.65:40297
I20260929 18:35:05.851673 11520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.65:40297 every 8 connection(s)
I20260929 18:35:05.852867 11371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
I20260929 18:35:05.857841 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 11371
I20260929 18:35:05.858352 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.66:32883
--local_ip_for_outbound_sockets=127.28.207.66
--tserver_master_addrs=127.28.207.126:43295
--webserver_port=35493
--webserver_interface=127.28.207.66
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:40155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:35:05.859588 11413 log.cc:824] T de4df53be70d4f61af6ab797ffc6289c P 7762bdb5642b48ceb9b2e3b59c82d078: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:05.867166 11413 tablet_bootstrap.cc:492] T de4df53be70d4f61af6ab797ffc6289c P 7762bdb5642b48ceb9b2e3b59c82d078: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:35:05.867653 11413 tablet_bootstrap.cc:492] T de4df53be70d4f61af6ab797ffc6289c P 7762bdb5642b48ceb9b2e3b59c82d078: Bootstrap complete.
Sep 29 18:35:05 dist-test-slave-qcpv krb5kdc[10751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706905, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:35:05.869315 11413 ts_tablet_manager.cc:1397] T de4df53be70d4f61af6ab797ffc6289c P 7762bdb5642b48ceb9b2e3b59c82d078: Time spent bootstrapping tablet: real 0.031s user 0.018s sys 0.010s
I20260929 18:35:05.872799 11413 raft_consensus.cc:348] T de4df53be70d4f61af6ab797ffc6289c P 7762bdb5642b48ceb9b2e3b59c82d078 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572967618bb4a0eb37fab57e504e8de" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 42373 } } peers { permanent_uuid: "7762bdb5642b48ceb9b2e3b59c82d078" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 40297 } } peers { permanent_uuid: "a0da9e3be899425d8c1d8b3bab353ea9" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 32883 } }
I20260929 18:35:05.873286 11413 raft_consensus.cc:729] T de4df53be70d4f61af6ab797ffc6289c P 7762bdb5642b48ceb9b2e3b59c82d078 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7762bdb5642b48ceb9b2e3b59c82d078, State: Initialized, Role: FOLLOWER
I20260929 18:35:05.873822 11413 consensus_queue.cc:260] T de4df53be70d4f61af6ab797ffc6289c P 7762bdb5642b48ceb9b2e3b59c82d078 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572967618bb4a0eb37fab57e504e8de" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 42373 } } peers { permanent_uuid: "7762bdb5642b48ceb9b2e3b59c82d078" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 40297 } } peers { permanent_uuid: "a0da9e3be899425d8c1d8b3bab353ea9" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 32883 } }
I20260929 18:35:05.874963 11413 ts_tablet_manager.cc:1428] T de4df53be70d4f61af6ab797ffc6289c P 7762bdb5642b48ceb9b2e3b59c82d078: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20260929 18:35:05.876428 11523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:05.856005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:35425 (local address 127.28.207.126:43295)
0929 18:35:05.856747 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:05.856777 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:05.857513 (+ 736us) server_negotiation.cc:407] Connection header received
0929 18:35:05.859197 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:05.859218 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:05.859766 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:05.860259 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:05.861876 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.863090 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.864773 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.865547 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.865746 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:05.869657 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:05.869665 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:05.869680 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:05.869727 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:05.872167 (+ 2440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:05.873264 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:05.873270 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:05.873276 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:05.873440 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:05.873964 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:05.873968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:05.873969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:05.874475 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:05.875291 (+ 816us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:05.875424 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 18:35:05.875703 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":245,"threads_started":1}
I20260929 18:35:05.876427 11524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:05.856000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.65:35425)
0929 18:35:05.857196 (+ 1196us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:05.857281 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:05.858458 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:05.860679 (+ 2221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:05.860698 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:05.861141 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:05.861664 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:05.861682 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.863302 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.863321 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:05.864621 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:05.864633 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.865757 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.865762 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:05.865998 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:05.866852 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:05.866914 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:05.869300 (+ 2386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:05.872436 (+ 3136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:05.872446 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:05.872470 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:05.873065 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:05.873565 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:05.873571 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:05.873574 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:05.873825 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:05.874629 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:05.874636 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:05.874968 (+ 332us) client_negotiation.cc:769] Sending connection context
0929 18:35:05.875176 (+ 208us) client_negotiation.cc:240] Negotiation successful
0929 18:35:05.875539 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":881,"thread_start_us":165,"threads_started":1}
I20260929 18:35:05.879324 11521 heartbeater.cc:347] Connected to a master server at 127.28.207.126:43295
I20260929 18:35:05.879583 11521 heartbeater.cc:464] Registering TS with master...
I20260929 18:35:05.880103 11521 heartbeater.cc:511] Master 127.28.207.126:43295 requested a full tablet report, sending...
I20260929 18:35:05.883186 11333 ts_manager.cc:194] Registered new tserver with Master: 7762bdb5642b48ceb9b2e3b59c82d078 (127.28.207.65:40297)
I20260929 18:35:05.888626 11333 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.65@KRBTEST.COM'} at 127.28.207.65:35425
I20260929 18:35:05.891832 11521 heartbeater.cc:503] Master 127.28.207.126:43295 was elected leader, sending a full tablet report...
Sep 29 18:35:05 dist-test-slave-qcpv krb5kdc[10751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706905, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for kudu/127.28.207.65@KRBTEST.COM
I20260929 18:35:05.901317 11528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:05.887992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:40297 (local address 127.0.0.1:57428)
0929 18:35:05.888934 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:05.888992 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:05.889225 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:05.889876 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:05.889883 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:05.890361 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:05.890510 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:05.890518 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.892117 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.892121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:05.892952 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:05.892961 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.893582 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.893586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:05.893697 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:05.894417 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:05.894429 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:05.896637 (+ 2208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:05.899264 (+ 2627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:05.899274 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:05.899281 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:05.899546 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:05.899896 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:05.899901 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:05.899903 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:05.899998 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:05.900550 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:05.900557 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:05.900705 (+ 148us) client_negotiation.cc:769] Sending connection context
0929 18:35:05.900867 (+ 162us) client_negotiation.cc:240] Negotiation successful
0929 18:35:05.901104 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"thread_start_us":162,"threads_started":1}
I20260929 18:35:05.901317 11529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:05.888158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57428 (local address 127.28.207.65:40297)
0929 18:35:05.888661 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:05.888673 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:05.889106 (+ 433us) server_negotiation.cc:407] Connection header received
0929 18:35:05.889388 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:05.889393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:05.889564 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:05.889718 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:05.890670 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.891918 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.893099 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:05.893461 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:05.893580 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:05.896838 (+ 3258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:05.896865 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:05.896877 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:05.896902 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:05.899083 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:05.899707 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:05.899713 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:05.899715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:05.899795 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:05.900117 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:05.900127 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:05.900129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:05.900414 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:05.900567 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:05.900955 (+ 388us) server_negotiation.cc:299] Negotiation successful
0929 18:35:05.901153 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":401,"thread_start_us":126,"threads_started":1}
I20260929 18:35:05.902906 11455 tablet_service.cc:1514] Processing DeleteTablet for tablet de4df53be70d4f61af6ab797ffc6289c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:05 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:57428
I20260929 18:35:05.904317 11530 tablet_replica.cc:330] stopping tablet replica
I20260929 18:35:05.904721 11530 raft_consensus.cc:2229] T de4df53be70d4f61af6ab797ffc6289c P 7762bdb5642b48ceb9b2e3b59c82d078 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:35:05.905025 11530 raft_consensus.cc:2258] T de4df53be70d4f61af6ab797ffc6289c P 7762bdb5642b48ceb9b2e3b59c82d078 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:35:05.906118 11530 ts_tablet_manager.cc:1905] T de4df53be70d4f61af6ab797ffc6289c P 7762bdb5642b48ceb9b2e3b59c82d078: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:35:05.910108 11530 ts_tablet_manager.cc:1918] T de4df53be70d4f61af6ab797ffc6289c P 7762bdb5642b48ceb9b2e3b59c82d078: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:35:05.910233 11530 log.cc:1196] T de4df53be70d4f61af6ab797ffc6289c P 7762bdb5642b48ceb9b2e3b59c82d078: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/wal/wals/de4df53be70d4f61af6ab797ffc6289c
I20260929 18:35:05.910503 11530 ts_tablet_manager.cc:1939] T de4df53be70d4f61af6ab797ffc6289c P 7762bdb5642b48ceb9b2e3b59c82d078: Deleting consensus metadata
I20260929 18:35:05.912515 11320 catalog_manager.cc:4841] TS 7762bdb5642b48ceb9b2e3b59c82d078 (127.28.207.65:40297): tablet de4df53be70d4f61af6ab797ffc6289c (table test-table [id=810d4080827f45e58f438201e946e151]) successfully deleted
W20260929 18:35:06.009173 11526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:06.009506 11526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:06.009553 11526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:06.013952 11526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:06.014029 11526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:06.014092 11526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.66
I20260929 18:35:06.019711 11526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:40155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.66:32883
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.207.66
--webserver_port=35493
--tserver_master_addrs=127.28.207.126:43295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.66
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:35:06.021096 11526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:06.022348 11526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:06.033876 11538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:06.034013 11526 server_base.cc:1034] running on GCE node
W20260929 18:35:06.033864 11536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:06.034214 11535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:06.034747 11526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:06.035588 11526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:06.036803 11526 hybrid_clock.cc:648] HybridClock initialized: now 1790706906036783 us; error 35 us; skew 500 ppm
Sep 29 18:35:06 dist-test-slave-qcpv krb5kdc[10751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706906, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:06.040155 11526 init.cc:377] Logged in from keytab as kudu/127.28.207.66@KRBTEST.COM (short username kudu)
I20260929 18:35:06.041286 11526 webserver.cc:460] Webserver started at http://127.28.207.66:35493/ using document root <none> and password file <none>
I20260929 18:35:06.041924 11526 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:06.042035 11526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:06.046406 11526 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:35:06.048975 11545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:06.050446 11526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:35:06.050607 11526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/wal
uuid: "a0da9e3be899425d8c1d8b3bab353ea9"
format_stamp: "Formatted at 2026-09-29 18:35:04 on dist-test-slave-qcpv"
server_key: "b1e05f7fc82c94fd4a24e39501dc47d6"
server_key_iv: "08d7667af4b3f8e91164defd6f9d55a4"
server_key_version: "encryptionkey@0"
I20260929 18:35:06.051333 11526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:06.066989 11526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:06.070694 11526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:06.070935 11526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:06.071632 11526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:35:06.072510 11526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:35:06.072588 11526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:06.072664 11526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:35:06.072707 11526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:06.083456 11526 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.66:32883
I20260929 18:35:06.083501 11658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.66:32883 every 8 connection(s)
I20260929 18:35:06.084640 11526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
I20260929 18:35:06.091753 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 11526
I20260929 18:35:06.092291 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.67:42373
--local_ip_for_outbound_sockets=127.28.207.67
--tserver_master_addrs=127.28.207.126:43295
--webserver_port=44107
--webserver_interface=127.28.207.67
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:40155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:35:06 dist-test-slave-qcpv krb5kdc[10751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706906, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:35:06.100883 11523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:06.087060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:35111 (local address 127.28.207.126:43295)
0929 18:35:06.087212 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:06.087216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:06.088438 (+ 1222us) server_negotiation.cc:407] Connection header received
0929 18:35:06.089937 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:06.089940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:06.090005 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:06.090111 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:06.091443 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.092046 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.093306 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.093548 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.093668 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:06.096863 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:06.096867 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:06.096868 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:06.096892 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:06.098517 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:06.099230 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:06.099234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:06.099235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:06.099322 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:06.099689 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:06.099693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:06.099696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:06.099885 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:06.099992 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:06.100523 (+ 531us) server_negotiation.cc:299] Negotiation successful
0929 18:35:06.100692 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:35:06.101732 11661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:06.087552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.66:35111)
0929 18:35:06.088179 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:06.088275 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:06.089698 (+ 1423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:06.090311 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:06.090318 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:06.090691 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:06.091268 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:06.091282 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.092190 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.092193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:06.093174 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:06.093185 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.093716 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.093720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:06.093901 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:06.094637 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:06.094653 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:06.096647 (+ 1994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:06.098648 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:06.098658 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:06.098674 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:06.099055 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:06.099418 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:06.099423 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:06.099427 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:06.099575 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:06.099990 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:06.099998 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:06.100276 (+ 278us) client_negotiation.cc:769] Sending connection context
0929 18:35:06.100470 (+ 194us) client_negotiation.cc:240] Negotiation successful
0929 18:35:06.100832 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":145,"threads_started":1}
I20260929 18:35:06.103178 11659 heartbeater.cc:347] Connected to a master server at 127.28.207.126:43295
I20260929 18:35:06.103457 11659 heartbeater.cc:464] Registering TS with master...
I20260929 18:35:06.103998 11659 heartbeater.cc:511] Master 127.28.207.126:43295 requested a full tablet report, sending...
I20260929 18:35:06.105237 11333 ts_manager.cc:194] Registered new tserver with Master: a0da9e3be899425d8c1d8b3bab353ea9 (127.28.207.66:32883)
I20260929 18:35:06.106151 11333 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.66@KRBTEST.COM'} at 127.28.207.66:35111
W20260929 18:35:06.227821 11662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:06.228183 11662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:06.228233 11662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:06.232633 11662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:06.232713 11662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:06.232786 11662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.67
I20260929 18:35:06.238566 11662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:40155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.67:42373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.207.67
--webserver_port=44107
--tserver_master_addrs=127.28.207.126:43295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.67
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:35:06.240049 11662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:06.241272 11662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:06.252943 11670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:06.253129 11662 server_base.cc:1034] running on GCE node
W20260929 18:35:06.252997 11667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:06.252997 11668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:06.253914 11662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:06.254683 11662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:06.255860 11662 hybrid_clock.cc:648] HybridClock initialized: now 1790706906255841 us; error 41 us; skew 500 ppm
Sep 29 18:35:06 dist-test-slave-qcpv krb5kdc[10751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706906, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:06.259001 11662 init.cc:377] Logged in from keytab as kudu/127.28.207.67@KRBTEST.COM (short username kudu)
I20260929 18:35:06.259966 11662 webserver.cc:460] Webserver started at http://127.28.207.67:44107/ using document root <none> and password file <none>
I20260929 18:35:06.260581 11662 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:06.260694 11662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:06.265084 11662 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:35:06.267851 11677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:06.269641 11662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:35:06.269913 11662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/wal
uuid: "a572967618bb4a0eb37fab57e504e8de"
format_stamp: "Formatted at 2026-09-29 18:35:04 on dist-test-slave-qcpv"
server_key: "ce4429dddf07e0305486a764b5b1e942"
server_key_iv: "35a2dc5d245391a98481adc2decf0b7b"
server_key_version: "encryptionkey@0"
I20260929 18:35:06.270601 11662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:06.293095 11662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:06.296449 11662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:06.296670 11662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:06.297300 11662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:35:06.298187 11662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:35:06.298261 11662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:06.298339 11662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:35:06.298384 11662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:06.309026 11662 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.67:42373
I20260929 18:35:06.309130 11790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.67:42373 every 8 connection(s)
I20260929 18:35:06.310138 11662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
I20260929 18:35:06.311352 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 11662
I20260929 18:35:06.322127 11523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:06.312165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55322 (local address 127.28.207.126:43295)
0929 18:35:06.312348 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:06.312354 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:06.312662 (+ 308us) server_negotiation.cc:407] Connection header received
0929 18:35:06.312936 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:06.312940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:06.313032 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:06.313144 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:06.314113 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.314696 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.315794 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.316028 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.316140 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:06.317892 (+ 1752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:06.317896 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:06.317898 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:06.317930 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:06.319882 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:06.320556 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:06.320560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:06.320562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:06.320700 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:06.321028 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:06.321031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:06.321033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:06.321310 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:06.321509 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:06.321796 (+ 287us) server_negotiation.cc:299] Negotiation successful
0929 18:35:06.321944 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 18:35:06 dist-test-slave-qcpv krb5kdc[10751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706906, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:35:06.328379 11794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:06.312599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:58487 (local address 127.28.207.126:43295)
0929 18:35:06.312933 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:06.312938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:06.313931 (+ 993us) server_negotiation.cc:407] Connection header received
0929 18:35:06.315381 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:06.315384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:06.315443 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:06.315556 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:06.317447 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.318117 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.320074 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.320507 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.320613 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:06.323990 (+ 3377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:06.323994 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:06.323997 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:06.324028 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:06.325742 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:06.326424 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:06.326429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:06.326431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:06.326517 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:06.327070 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:06.327075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:06.327081 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:06.327288 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:06.327417 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:06.328032 (+ 615us) server_negotiation.cc:299] Negotiation successful
0929 18:35:06.328175 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"spinlock_wait_cycles":1408,"thread_start_us":87,"threads_started":1}
I20260929 18:35:06.329433 11795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:06.313203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.67:58487)
0929 18:35:06.313727 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:06.313795 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:06.315166 (+ 1371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:06.315824 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:06.315832 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:06.316485 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:06.317265 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:06.317281 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.318266 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.318272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:06.319912 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:06.319926 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.320709 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.320714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:06.320945 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:06.321678 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:06.321720 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:06.323781 (+ 2061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:06.325946 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:06.325960 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:06.325979 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:06.326298 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:06.326616 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:06.326620 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:06.326622 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:06.326784 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:06.327408 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:06.327416 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:06.327752 (+ 336us) client_negotiation.cc:769] Sending connection context
0929 18:35:06.327971 (+ 219us) client_negotiation.cc:240] Negotiation successful
0929 18:35:06.328389 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":141,"threads_started":1}
I20260929 18:35:06.331130 11791 heartbeater.cc:347] Connected to a master server at 127.28.207.126:43295
I20260929 18:35:06.331414 11791 heartbeater.cc:464] Registering TS with master...
I20260929 18:35:06.332022 11791 heartbeater.cc:511] Master 127.28.207.126:43295 requested a full tablet report, sending...
I20260929 18:35:06.333582 11333 ts_manager.cc:194] Registered new tserver with Master: a572967618bb4a0eb37fab57e504e8de (127.28.207.67:42373)
I20260929 18:35:06.334295 11333 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.67@KRBTEST.COM'} at 127.28.207.67:58487
I20260929 18:35:06.336534 29501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:35:06.341913 11794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:06.337387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55324 (local address 127.28.207.126:43295)
0929 18:35:06.337561 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:06.337567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:06.337841 (+ 274us) server_negotiation.cc:407] Connection header received
0929 18:35:06.338003 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:06.338005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:06.338117 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:06.338222 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:35:06.338564 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.339215 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.340184 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.340431 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.340531 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:06.340887 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:35:06.341384 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:35:06.341494 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:06.341623 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 18:35:06.341691 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 18:35:06.342757 11333 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:55324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:35:06.346693 11333 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:35:06.356673 11455 tablet_service.cc:1467] Processing CreateTablet for tablet fac673c0b4db4b8bb151f0bf9477d07b (DEFAULT_TABLE table=test-table [id=e810a0191846492b9bb4ac382b078fb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:06.357440 11455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fac673c0b4db4b8bb151f0bf9477d07b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:06.360745 11413 tablet_bootstrap.cc:492] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078: Bootstrap starting.
I20260929 18:35:06.361752 11413 tablet_bootstrap.cc:654] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:06.362141 11528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:06.355506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:42373 (local address 127.0.0.1:41994)
0929 18:35:06.356178 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:06.356197 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:06.356292 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:06.357723 (+ 1431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:06.357726 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:06.357746 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:06.357926 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:06.357934 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.359322 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.359328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:06.360679 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:06.360692 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.361638 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.361642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:06.361743 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:06.361799 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 18:35:06.361877 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 18:35:06.361979 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"mutex_wait_us":3}
I20260929 18:35:06.362161 11802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:06.356502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41994 (local address 127.28.207.67:42373)
0929 18:35:06.357054 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:06.357069 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:06.357106 (+ 37us) server_negotiation.cc:407] Connection header received
0929 18:35:06.357191 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:06.357196 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:06.357427 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:06.358100 (+ 673us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:06.358278 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.359177 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.360833 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.361493 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.361627 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:06.361805 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:06.361919 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 18:35:06.362023 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":189,"threads_started":1}
I20260929 18:35:06.363103 11800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:06.355833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:32883 (local address 127.0.0.1:41576)
0929 18:35:06.356201 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:06.356219 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:06.356330 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:06.357937 (+ 1607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:06.357940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:06.357957 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:06.358060 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:06.358066 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.359567 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.359571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:06.360692 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:06.360698 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.362658 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.362661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:06.362751 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:06.362786 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 18:35:06.362898 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 18:35:06.362972 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":31,"thread_start_us":154,"threads_started":1}
I20260929 18:35:06.363209 11801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:06.356028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41576 (local address 127.28.207.66:32883)
0929 18:35:06.357288 (+ 1260us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:06.357302 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:06.357335 (+ 33us) server_negotiation.cc:407] Connection header received
0929 18:35:06.357429 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:06.357435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:06.357755 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:06.357963 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:06.358183 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.359398 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.360860 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.361758 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.361934 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:06.362130 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:06.362942 (+ 812us) server_negotiation.cc:299] Negotiation successful
0929 18:35:06.363055 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1119,"thread_start_us":162,"threads_started":1}
I20260929 18:35:06.364288 11413 tablet_bootstrap.cc:492] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078: No bootstrap required, opened a new log
I20260929 18:35:06.364387 11413 ts_tablet_manager.cc:1397] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260929 18:35:06.364847 11413 raft_consensus.cc:348] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572967618bb4a0eb37fab57e504e8de" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 42373 } } peers { permanent_uuid: "a0da9e3be899425d8c1d8b3bab353ea9" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 32883 } } peers { permanent_uuid: "7762bdb5642b48ceb9b2e3b59c82d078" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 40297 } }
I20260929 18:35:06.364982 11413 raft_consensus.cc:374] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:06.365020 11413 raft_consensus.cc:729] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7762bdb5642b48ceb9b2e3b59c82d078, State: Initialized, Role: FOLLOWER
I20260929 18:35:06.365172 11413 consensus_queue.cc:260] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572967618bb4a0eb37fab57e504e8de" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 42373 } } peers { permanent_uuid: "a0da9e3be899425d8c1d8b3bab353ea9" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 32883 } } peers { permanent_uuid: "7762bdb5642b48ceb9b2e3b59c82d078" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 40297 } }
I20260929 18:35:06.365926 11413 ts_tablet_manager.cc:1428] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 18:35:06.366065 11725 tablet_service.cc:1467] Processing CreateTablet for tablet fac673c0b4db4b8bb151f0bf9477d07b (DEFAULT_TABLE table=test-table [id=e810a0191846492b9bb4ac382b078fb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:06.366521 11593 tablet_service.cc:1467] Processing CreateTablet for tablet fac673c0b4db4b8bb151f0bf9477d07b (DEFAULT_TABLE table=test-table [id=e810a0191846492b9bb4ac382b078fb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:06.367539 11725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fac673c0b4db4b8bb151f0bf9477d07b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:06.367681 11593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fac673c0b4db4b8bb151f0bf9477d07b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:06.375717 11804 tablet_bootstrap.cc:492] T fac673c0b4db4b8bb151f0bf9477d07b P a0da9e3be899425d8c1d8b3bab353ea9: Bootstrap starting.
I20260929 18:35:06.376617 11805 tablet_bootstrap.cc:492] T fac673c0b4db4b8bb151f0bf9477d07b P a572967618bb4a0eb37fab57e504e8de: Bootstrap starting.
I20260929 18:35:06.379154 11804 tablet_bootstrap.cc:654] T fac673c0b4db4b8bb151f0bf9477d07b P a0da9e3be899425d8c1d8b3bab353ea9: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:06.379457 11805 tablet_bootstrap.cc:654] T fac673c0b4db4b8bb151f0bf9477d07b P a572967618bb4a0eb37fab57e504e8de: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:06.380105 11804 log.cc:824] T fac673c0b4db4b8bb151f0bf9477d07b P a0da9e3be899425d8c1d8b3bab353ea9: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:06.380309 11805 log.cc:824] T fac673c0b4db4b8bb151f0bf9477d07b P a572967618bb4a0eb37fab57e504e8de: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:06.382393 11804 tablet_bootstrap.cc:492] T fac673c0b4db4b8bb151f0bf9477d07b P a0da9e3be899425d8c1d8b3bab353ea9: No bootstrap required, opened a new log
I20260929 18:35:06.382428 11805 tablet_bootstrap.cc:492] T fac673c0b4db4b8bb151f0bf9477d07b P a572967618bb4a0eb37fab57e504e8de: No bootstrap required, opened a new log
I20260929 18:35:06.382643 11804 ts_tablet_manager.cc:1397] T fac673c0b4db4b8bb151f0bf9477d07b P a0da9e3be899425d8c1d8b3bab353ea9: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260929 18:35:06.382663 11805 ts_tablet_manager.cc:1397] T fac673c0b4db4b8bb151f0bf9477d07b P a572967618bb4a0eb37fab57e504e8de: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260929 18:35:06.386189 11804 raft_consensus.cc:348] T fac673c0b4db4b8bb151f0bf9477d07b P a0da9e3be899425d8c1d8b3bab353ea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572967618bb4a0eb37fab57e504e8de" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 42373 } } peers { permanent_uuid: "a0da9e3be899425d8c1d8b3bab353ea9" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 32883 } } peers { permanent_uuid: "7762bdb5642b48ceb9b2e3b59c82d078" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 40297 } }
I20260929 18:35:06.386447 11804 raft_consensus.cc:374] T fac673c0b4db4b8bb151f0bf9477d07b P a0da9e3be899425d8c1d8b3bab353ea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:06.386513 11804 raft_consensus.cc:729] T fac673c0b4db4b8bb151f0bf9477d07b P a0da9e3be899425d8c1d8b3bab353ea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0da9e3be899425d8c1d8b3bab353ea9, State: Initialized, Role: FOLLOWER
I20260929 18:35:06.386667 11805 raft_consensus.cc:348] T fac673c0b4db4b8bb151f0bf9477d07b P a572967618bb4a0eb37fab57e504e8de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572967618bb4a0eb37fab57e504e8de" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 42373 } } peers { permanent_uuid: "a0da9e3be899425d8c1d8b3bab353ea9" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 32883 } } peers { permanent_uuid: "7762bdb5642b48ceb9b2e3b59c82d078" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 40297 } }
I20260929 18:35:06.386993 11805 raft_consensus.cc:374] T fac673c0b4db4b8bb151f0bf9477d07b P a572967618bb4a0eb37fab57e504e8de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:06.387063 11805 raft_consensus.cc:729] T fac673c0b4db4b8bb151f0bf9477d07b P a572967618bb4a0eb37fab57e504e8de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a572967618bb4a0eb37fab57e504e8de, State: Initialized, Role: FOLLOWER
I20260929 18:35:06.387076 11804 consensus_queue.cc:260] T fac673c0b4db4b8bb151f0bf9477d07b P a0da9e3be899425d8c1d8b3bab353ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572967618bb4a0eb37fab57e504e8de" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 42373 } } peers { permanent_uuid: "a0da9e3be899425d8c1d8b3bab353ea9" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 32883 } } peers { permanent_uuid: "7762bdb5642b48ceb9b2e3b59c82d078" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 40297 } }
I20260929 18:35:06.387545 11805 consensus_queue.cc:260] T fac673c0b4db4b8bb151f0bf9477d07b P a572967618bb4a0eb37fab57e504e8de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572967618bb4a0eb37fab57e504e8de" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 42373 } } peers { permanent_uuid: "a0da9e3be899425d8c1d8b3bab353ea9" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 32883 } } peers { permanent_uuid: "7762bdb5642b48ceb9b2e3b59c82d078" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 40297 } }
I20260929 18:35:06.387825 11659 heartbeater.cc:503] Master 127.28.207.126:43295 was elected leader, sending a full tablet report...
I20260929 18:35:06.388224 11804 ts_tablet_manager.cc:1428] T fac673c0b4db4b8bb151f0bf9477d07b P a0da9e3be899425d8c1d8b3bab353ea9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 18:35:06.388312 11791 heartbeater.cc:503] Master 127.28.207.126:43295 was elected leader, sending a full tablet report...
I20260929 18:35:06.388628 11805 ts_tablet_manager.cc:1428] T fac673c0b4db4b8bb151f0bf9477d07b P a572967618bb4a0eb37fab57e504e8de: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 18:35:06.489279 11809 raft_consensus.cc:482] T fac673c0b4db4b8bb151f0bf9477d07b P a572967618bb4a0eb37fab57e504e8de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:35:06.489516 11809 raft_consensus.cc:504] T fac673c0b4db4b8bb151f0bf9477d07b P a572967618bb4a0eb37fab57e504e8de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572967618bb4a0eb37fab57e504e8de" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 42373 } } peers { permanent_uuid: "a0da9e3be899425d8c1d8b3bab353ea9" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 32883 } } peers { permanent_uuid: "7762bdb5642b48ceb9b2e3b59c82d078" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 40297 } }
I20260929 18:35:06.490902 11809 leader_election.cc:288] T fac673c0b4db4b8bb151f0bf9477d07b P a572967618bb4a0eb37fab57e504e8de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0da9e3be899425d8c1d8b3bab353ea9 (127.28.207.66:32883), 7762bdb5642b48ceb9b2e3b59c82d078 (127.28.207.65:40297)
I20260929 18:35:06.495584 11801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:06.491241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:36759 (local address 127.28.207.66:32883)
0929 18:35:06.491394 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:06.491400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:06.491445 (+ 45us) server_negotiation.cc:407] Connection header received
0929 18:35:06.491673 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:06.491679 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:06.491748 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:06.491851 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:06.492239 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.492907 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.494331 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.494958 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.495085 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:06.495130 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:06.495424 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 18:35:06.495482 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 18:35:06.495674 11795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:06.491140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:32883 (local address 127.28.207.67:36759)
0929 18:35:06.491355 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:06.491375 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:06.491503 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:06.491903 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:06.491907 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:06.491930 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:06.492093 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:06.492104 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.493041 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.493045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:06.494196 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:06.494206 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.495115 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.495119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:06.495269 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:06.495302 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 18:35:06.495401 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 18:35:06.495492 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":26}
I20260929 18:35:06.496464 11811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:06.491427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:40297 (local address 127.28.207.67:44807)
0929 18:35:06.491987 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:06.492001 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:06.492083 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:06.492429 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:06.492433 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:06.492452 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:06.492590 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:06.492597 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.493702 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.493704 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:06.494904 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:06.494915 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.495964 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.495967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:06.496125 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:06.496159 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 18:35:06.496242 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 18:35:06.496322 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":23,"thread_start_us":255,"threads_started":1}
I20260929 18:35:06.496575 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:06.491546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:44807 (local address 127.28.207.65:40297)
0929 18:35:06.492080 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:06.492091 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:06.492109 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:35:06.492174 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:06.492179 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:06.492272 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:06.492388 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:06.492752 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.493569 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.495094 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.495796 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.495925 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:06.496104 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:06.496335 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 18:35:06.496433 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":112,"threads_started":1}
I20260929 18:35:06.496748 11613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fac673c0b4db4b8bb151f0bf9477d07b" candidate_uuid: "a572967618bb4a0eb37fab57e504e8de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0da9e3be899425d8c1d8b3bab353ea9" is_pre_election: true
I20260929 18:35:06.497190 11613 raft_consensus.cc:2454] T fac673c0b4db4b8bb151f0bf9477d07b P a0da9e3be899425d8c1d8b3bab353ea9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a572967618bb4a0eb37fab57e504e8de in term 0.
I20260929 18:35:06.497236 11474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fac673c0b4db4b8bb151f0bf9477d07b" candidate_uuid: "a572967618bb4a0eb37fab57e504e8de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7762bdb5642b48ceb9b2e3b59c82d078" is_pre_election: true
I20260929 18:35:06.497573 11474 raft_consensus.cc:2454] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a572967618bb4a0eb37fab57e504e8de in term 0.
I20260929 18:35:06.497768 11678 leader_election.cc:302] T fac673c0b4db4b8bb151f0bf9477d07b P a572967618bb4a0eb37fab57e504e8de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0da9e3be899425d8c1d8b3bab353ea9, a572967618bb4a0eb37fab57e504e8de; no voters:
I20260929 18:35:06.498147 11809 raft_consensus.cc:2781] T fac673c0b4db4b8bb151f0bf9477d07b P a572967618bb4a0eb37fab57e504e8de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:35:06.498240 11809 raft_consensus.cc:482] T fac673c0b4db4b8bb151f0bf9477d07b P a572967618bb4a0eb37fab57e504e8de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:35:06.498272 11809 raft_consensus.cc:3037] T fac673c0b4db4b8bb151f0bf9477d07b P a572967618bb4a0eb37fab57e504e8de [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:06.499886 11809 raft_consensus.cc:504] T fac673c0b4db4b8bb151f0bf9477d07b P a572967618bb4a0eb37fab57e504e8de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572967618bb4a0eb37fab57e504e8de" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 42373 } } peers { permanent_uuid: "a0da9e3be899425d8c1d8b3bab353ea9" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 32883 } } peers { permanent_uuid: "7762bdb5642b48ceb9b2e3b59c82d078" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 40297 } }
I20260929 18:35:06.500501 11809 leader_election.cc:288] T fac673c0b4db4b8bb151f0bf9477d07b P a572967618bb4a0eb37fab57e504e8de [CANDIDATE]: Term 1 election: Requested vote from peers a0da9e3be899425d8c1d8b3bab353ea9 (127.28.207.66:32883), 7762bdb5642b48ceb9b2e3b59c82d078 (127.28.207.65:40297)
I20260929 18:35:06.501037 11613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fac673c0b4db4b8bb151f0bf9477d07b" candidate_uuid: "a572967618bb4a0eb37fab57e504e8de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0da9e3be899425d8c1d8b3bab353ea9"
I20260929 18:35:06.501044 11474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fac673c0b4db4b8bb151f0bf9477d07b" candidate_uuid: "a572967618bb4a0eb37fab57e504e8de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7762bdb5642b48ceb9b2e3b59c82d078"
I20260929 18:35:06.501199 11613 raft_consensus.cc:3037] T fac673c0b4db4b8bb151f0bf9477d07b P a0da9e3be899425d8c1d8b3bab353ea9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:06.501199 11474 raft_consensus.cc:3037] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:06.502693 11474 raft_consensus.cc:2454] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a572967618bb4a0eb37fab57e504e8de in term 1.
I20260929 18:35:06.502693 11613 raft_consensus.cc:2454] T fac673c0b4db4b8bb151f0bf9477d07b P a0da9e3be899425d8c1d8b3bab353ea9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a572967618bb4a0eb37fab57e504e8de in term 1.
I20260929 18:35:06.503262 11681 leader_election.cc:302] T fac673c0b4db4b8bb151f0bf9477d07b P a572967618bb4a0eb37fab57e504e8de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7762bdb5642b48ceb9b2e3b59c82d078, a0da9e3be899425d8c1d8b3bab353ea9, a572967618bb4a0eb37fab57e504e8de; no voters:
I20260929 18:35:06.503491 11809 raft_consensus.cc:2781] T fac673c0b4db4b8bb151f0bf9477d07b P a572967618bb4a0eb37fab57e504e8de [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:06.503767 11809 raft_consensus.cc:686] T fac673c0b4db4b8bb151f0bf9477d07b P a572967618bb4a0eb37fab57e504e8de [term 1 LEADER]: Becoming Leader. State: Replica: a572967618bb4a0eb37fab57e504e8de, State: Running, Role: LEADER
I20260929 18:35:06.504197 11809 consensus_queue.cc:237] T fac673c0b4db4b8bb151f0bf9477d07b P a572967618bb4a0eb37fab57e504e8de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572967618bb4a0eb37fab57e504e8de" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 42373 } } peers { permanent_uuid: "a0da9e3be899425d8c1d8b3bab353ea9" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 32883 } } peers { permanent_uuid: "7762bdb5642b48ceb9b2e3b59c82d078" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 40297 } }
I20260929 18:35:06.508507 11331 catalog_manager.cc:5495] T fac673c0b4db4b8bb151f0bf9477d07b P a572967618bb4a0eb37fab57e504e8de reported cstate change: term changed from 0 to 1, leader changed from <none> to a572967618bb4a0eb37fab57e504e8de (127.28.207.67). New cstate: current_term: 1 leader_uuid: "a572967618bb4a0eb37fab57e504e8de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572967618bb4a0eb37fab57e504e8de" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 42373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a0da9e3be899425d8c1d8b3bab353ea9" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 32883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7762bdb5642b48ceb9b2e3b59c82d078" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 40297 } health_report { overall_health: UNKNOWN } } }
I20260929 18:35:06.526103 11802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:06.522238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42006 (local address 127.28.207.67:42373)
0929 18:35:06.522389 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:06.522395 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:06.522410 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:35:06.522516 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:06.522519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:06.522574 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:06.522697 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:35:06.522974 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.523494 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.524532 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:06.524758 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:06.524872 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:06.525238 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:35:06.525685 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:35:06.525826 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:06.525888 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 18:35:06.525948 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:35:06.533448 11474 raft_consensus.cc:1261] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [term 1 FOLLOWER]: Refusing update from remote peer a572967618bb4a0eb37fab57e504e8de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:35:06.533578 11613 raft_consensus.cc:1261] T fac673c0b4db4b8bb151f0bf9477d07b P a0da9e3be899425d8c1d8b3bab353ea9 [term 1 FOLLOWER]: Refusing update from remote peer a572967618bb4a0eb37fab57e504e8de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:35:06.534472 11812 consensus_queue.cc:1035] T fac673c0b4db4b8bb151f0bf9477d07b P a572967618bb4a0eb37fab57e504e8de [LEADER]: Connected to new peer: Peer: permanent_uuid: "7762bdb5642b48ceb9b2e3b59c82d078" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:35:06.534739 11809 consensus_queue.cc:1035] T fac673c0b4db4b8bb151f0bf9477d07b P a572967618bb4a0eb37fab57e504e8de [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0da9e3be899425d8c1d8b3bab353ea9" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 32883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:35:06.542093 11815 mvcc.cc:205] Tried to move back new op lower bound from 7334735489151393792 to 7334735489044496384. Current Snapshot: MvccSnapshot[applied={T|T < 7334735489151393792}]
I20260929 18:35:06.542666 11816 mvcc.cc:205] Tried to move back new op lower bound from 7334735489151393792 to 7334735489044496384. Current Snapshot: MvccSnapshot[applied={T|T < 7334735489151393792}]
I20260929 18:35:06.544008 11817 mvcc.cc:205] Tried to move back new op lower bound from 7334735489151393792 to 7334735489044496384. Current Snapshot: MvccSnapshot[applied={T|T < 7334735489151393792}]
I20260929 18:35:06.553349 11331 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:55324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:35:06.553570 11331 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:55324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:35:06.556492 11331 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036: Sending DeleteTablet for 3 replicas of tablet fac673c0b4db4b8bb151f0bf9477d07b
I20260929 18:35:06.557539 11725 tablet_service.cc:1514] Processing DeleteTablet for tablet fac673c0b4db4b8bb151f0bf9477d07b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:06 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:41994
I20260929 18:35:06.557641 11593 tablet_service.cc:1514] Processing DeleteTablet for tablet fac673c0b4db4b8bb151f0bf9477d07b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:06 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:41576
I20260929 18:35:06.558163 11455 tablet_service.cc:1514] Processing DeleteTablet for tablet fac673c0b4db4b8bb151f0bf9477d07b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:06 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:57428
I20260929 18:35:06.558244 11825 tablet_replica.cc:330] T fac673c0b4db4b8bb151f0bf9477d07b P a0da9e3be899425d8c1d8b3bab353ea9: stopping tablet replica
I20260929 18:35:06.558601 11827 tablet_replica.cc:330] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078: stopping tablet replica
I20260929 18:35:06.558661 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 11371
I20260929 18:35:06.558743 11825 raft_consensus.cc:2229] T fac673c0b4db4b8bb151f0bf9477d07b P a0da9e3be899425d8c1d8b3bab353ea9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:35:06.558964 11827 raft_consensus.cc:2229] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:35:06.559206 11825 raft_consensus.cc:2258] T fac673c0b4db4b8bb151f0bf9477d07b P a0da9e3be899425d8c1d8b3bab353ea9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:35:06.560906 11825 ts_tablet_manager.cc:1905] T fac673c0b4db4b8bb151f0bf9477d07b P a0da9e3be899425d8c1d8b3bab353ea9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:35:06.563064 11826 tablet_replica.cc:330] T fac673c0b4db4b8bb151f0bf9477d07b P a572967618bb4a0eb37fab57e504e8de: stopping tablet replica
I20260929 18:35:06.563504 11826 raft_consensus.cc:2229] T fac673c0b4db4b8bb151f0bf9477d07b P a572967618bb4a0eb37fab57e504e8de [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:35:06.564134 11826 raft_consensus.cc:2258] T fac673c0b4db4b8bb151f0bf9477d07b P a572967618bb4a0eb37fab57e504e8de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:35:06.564177 11825 ts_tablet_manager.cc:1918] T fac673c0b4db4b8bb151f0bf9477d07b P a0da9e3be899425d8c1d8b3bab353ea9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:35:06.564302 11825 log.cc:1196] T fac673c0b4db4b8bb151f0bf9477d07b P a0da9e3be899425d8c1d8b3bab353ea9: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/wal/wals/fac673c0b4db4b8bb151f0bf9477d07b
I20260929 18:35:06.564922 11825 ts_tablet_manager.cc:1939] T fac673c0b4db4b8bb151f0bf9477d07b P a0da9e3be899425d8c1d8b3bab353ea9: Deleting consensus metadata
I20260929 18:35:06.566183 11826 ts_tablet_manager.cc:1905] T fac673c0b4db4b8bb151f0bf9477d07b P a572967618bb4a0eb37fab57e504e8de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:35:06.566558 11317 catalog_manager.cc:4841] TS a0da9e3be899425d8c1d8b3bab353ea9 (127.28.207.66:32883): tablet fac673c0b4db4b8bb151f0bf9477d07b (table test-table [id=e810a0191846492b9bb4ac382b078fb3]) successfully deleted
I20260929 18:35:06.569195 11826 ts_tablet_manager.cc:1918] T fac673c0b4db4b8bb151f0bf9477d07b P a572967618bb4a0eb37fab57e504e8de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:35:06.569332 11826 log.cc:1196] T fac673c0b4db4b8bb151f0bf9477d07b P a572967618bb4a0eb37fab57e504e8de: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/wal/wals/fac673c0b4db4b8bb151f0bf9477d07b
I20260929 18:35:06.569727 11826 ts_tablet_manager.cc:1939] T fac673c0b4db4b8bb151f0bf9477d07b P a572967618bb4a0eb37fab57e504e8de: Deleting consensus metadata
W20260929 18:35:06.570384 11320 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.207.65:40297 (error 108)
I20260929 18:35:06.570972 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 11526
I20260929 18:35:06.572788 11800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:06.572488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:40297 (local address 127.0.0.1:57442)
0929 18:35:06.572630 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:06.572700 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.65:40297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 18:35:06.573027 11320 catalog_manager.cc:4584] TS 7762bdb5642b48ceb9b2e3b59c82d078 (127.28.207.65:40297): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fac673c0b4db4b8bb151f0bf9477d07b: Network error: Client connection negotiation failed: client connection to 127.28.207.65:40297: connect: Connection refused (error 111)
I20260929 18:35:06.574649 11319 catalog_manager.cc:4841] TS a572967618bb4a0eb37fab57e504e8de (127.28.207.67:42373): tablet fac673c0b4db4b8bb151f0bf9477d07b (table test-table [id=e810a0191846492b9bb4ac382b078fb3]) successfully deleted
I20260929 18:35:06.580579 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 11662
I20260929 18:35:06.586145 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 11300
I20260929 18:35:06.592523 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:43295
--webserver_interface=127.28.207.126
--webserver_port=41305
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:40155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:43295
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:06.726094 11829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:06.726452 11829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:06.726501 11829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:06.731001 11829 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:35:06.731091 11829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:06.731123 11829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:06.731141 11829 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:35:06.731158 11829 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:35:06.737349 11829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:40155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:43295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:43295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=41305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:35:06.738921 11829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:06.740236 11829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:06.750258 11834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:06.750298 11837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:06.750433 11829 server_base.cc:1034] running on GCE node
W20260929 18:35:06.750800 11835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:06.751518 11829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:06.752919 11829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:06.754130 11829 hybrid_clock.cc:648] HybridClock initialized: now 1790706906754090 us; error 57 us; skew 500 ppm
Sep 29 18:35:06 dist-test-slave-qcpv krb5kdc[10751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706906, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:06.757506 11829 init.cc:377] Logged in from keytab as kudu/127.28.207.126@KRBTEST.COM (short username kudu)
I20260929 18:35:06.758665 11829 webserver.cc:460] Webserver started at http://127.28.207.126:41305/ using document root <none> and password file <none>
I20260929 18:35:06.759424 11829 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:06.759527 11829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:06.764599 11829 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:35:06.767313 11844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:06.769055 11829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 18:35:06.769234 11829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "bf5d34d45bd44585ac555847b1786036"
format_stamp: "Formatted at 2026-09-29 18:35:04 on dist-test-slave-qcpv"
server_key: "c58ab8fc1bc9514d7182bb0d2c81a373"
server_key_iv: "c619fbd511652c0d466eef07620da584"
server_key_version: "encryptionkey@0"
I20260929 18:35:06.769910 11829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:06.790732 11829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:06.794391 11829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:06.794643 11829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:06.803611 11829 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:43295
I20260929 18:35:06.803968 11896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:43295 every 8 connection(s)
I20260929 18:35:06.804977 11829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:35:06.809639 11897 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:35:06.810734 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 11829
I20260929 18:35:06.811270 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.65:40297
--local_ip_for_outbound_sockets=127.28.207.65
--tserver_master_addrs=127.28.207.126:43295
--webserver_port=35325
--webserver_interface=127.28.207.65
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:40155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:35:06.811618 11897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036: Bootstrap starting.
I20260929 18:35:06.830312 11897 log.cc:824] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:06.841732 11897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:35:06.842176 11897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036: Bootstrap complete.
I20260929 18:35:06.846349 11897 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5d34d45bd44585ac555847b1786036" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43295 } }
I20260929 18:35:06.846762 11897 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf5d34d45bd44585ac555847b1786036, State: Initialized, Role: FOLLOWER
I20260929 18:35:06.847337 11897 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5d34d45bd44585ac555847b1786036" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43295 } }
I20260929 18:35:06.847461 11897 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:35:06.847553 11897 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:35:06.847671 11897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [term 2 FOLLOWER]: Advancing to term 3
I20260929 18:35:06.849646 11897 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5d34d45bd44585ac555847b1786036" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43295 } }
I20260929 18:35:06.849983 11897 leader_election.cc:302] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf5d34d45bd44585ac555847b1786036; no voters:
I20260929 18:35:06.850308 11897 leader_election.cc:288] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 18:35:06.850592 11901 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 18:35:06.850978 11901 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [term 3 LEADER]: Becoming Leader. State: Replica: bf5d34d45bd44585ac555847b1786036, State: Running, Role: LEADER
I20260929 18:35:06.851399 11901 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5d34d45bd44585ac555847b1786036" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43295 } }
I20260929 18:35:06.851883 11897 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:35:06.853107 11902 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "bf5d34d45bd44585ac555847b1786036" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5d34d45bd44585ac555847b1786036" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43295 } } }
I20260929 18:35:06.853259 11902 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:06.854206 11910 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:35:06.854602 11903 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf5d34d45bd44585ac555847b1786036. Latest consensus state: current_term: 3 leader_uuid: "bf5d34d45bd44585ac555847b1786036" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5d34d45bd44585ac555847b1786036" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43295 } } }
I20260929 18:35:06.855156 11903 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf5d34d45bd44585ac555847b1786036 [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:06.859463 11910 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:35:06.860129 11910 catalog_manager.cc:1259] Loaded cluster ID: 0994ef4f2f244e3b854dd0340735201e
I20260929 18:35:06.860191 11910 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 18:35:06.861198 11910 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f93fc63b9ad google::LogMessage::Fail() at ??:0
@ 0x7f93fc63e88a google::LogMessage::SendToLog() at ??:0
@ 0x7f93fc63b4de google::LogMessage::Flush() at ??:0
@ 0x7f93fc63ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9406f7afdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f9406f7bd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9406f78579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f9406fbc640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f93fe88d3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f93fe88dc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f93fe88f3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f93fe87e512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f93ffec26db start_thread at ??:0
@ 0x7f93fb13961f clone at ??:0
W20260929 18:35:06.975348 11899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:06.975677 11899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:06.975728 11899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:06.980042 11899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:06.980114 11899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:06.980203 11899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.65
I20260929 18:35:06.985814 11899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:40155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.65:40297
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.207.65
--webserver_port=35325
--tserver_master_addrs=127.28.207.126:43295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.65
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:35:06.987226 11899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:06.988660 11899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:07.001269 11939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:07.001269 11938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:07.001549 11941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:07.001919 11899 server_base.cc:1034] running on GCE node
I20260929 18:35:07.002625 11899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:07.003631 11899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:07.004860 11899 hybrid_clock.cc:648] HybridClock initialized: now 1790706907004814 us; error 61 us; skew 500 ppm
Sep 29 18:35:07 dist-test-slave-qcpv krb5kdc[10751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706907, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:07.008700 11899 init.cc:377] Logged in from keytab as kudu/127.28.207.65@KRBTEST.COM (short username kudu)
I20260929 18:35:07.010313 11899 webserver.cc:460] Webserver started at http://127.28.207.65:35325/ using document root <none> and password file <none>
I20260929 18:35:07.011337 11899 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:07.011449 11899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:07.016880 11899 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260929 18:35:07.019924 11948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:07.021775 11899 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260929 18:35:07.021996 11899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/wal
uuid: "7762bdb5642b48ceb9b2e3b59c82d078"
format_stamp: "Formatted at 2026-09-29 18:35:04 on dist-test-slave-qcpv"
server_key: "92dc09a9b2a3ece0a264009f308d9b30"
server_key_iv: "3a1b1d59037ff84b88b22cd467354602"
server_key_version: "encryptionkey@0"
I20260929 18:35:07.022977 11899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:07.037909 11899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:07.042672 11899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:07.042964 11899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:07.043864 11899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:35:07.045517 11955 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 18:35:07.050164 11899 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 18:35:07.050264 11899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.001s
I20260929 18:35:07.050347 11899 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 18:35:07.053333 11899 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 18:35:07.053419 11899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260929 18:35:07.053893 11955 tablet_bootstrap.cc:492] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078: Bootstrap starting.
I20260929 18:35:07.067350 11899 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.65:40297
I20260929 18:35:07.068892 11899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
I20260929 18:35:07.070461 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 11899
I20260929 18:35:07.070981 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.66:32883
--local_ip_for_outbound_sockets=127.28.207.66
--tserver_master_addrs=127.28.207.126:43295
--webserver_port=35493
--webserver_interface=127.28.207.66
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:40155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:35:07.071600 12062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.65:40297 every 8 connection(s)
I20260929 18:35:07.075881 11955 log.cc:824] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:07.086787 11955 tablet_bootstrap.cc:492] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:35:07.087308 11955 tablet_bootstrap.cc:492] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078: Bootstrap complete.
I20260929 18:35:07.088793 11955 ts_tablet_manager.cc:1397] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078: Time spent bootstrapping tablet: real 0.035s user 0.026s sys 0.006s
I20260929 18:35:07.092402 11955 raft_consensus.cc:348] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572967618bb4a0eb37fab57e504e8de" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 42373 } } peers { permanent_uuid: "a0da9e3be899425d8c1d8b3bab353ea9" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 32883 } } peers { permanent_uuid: "7762bdb5642b48ceb9b2e3b59c82d078" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 40297 } }
I20260929 18:35:07.092969 11955 raft_consensus.cc:729] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7762bdb5642b48ceb9b2e3b59c82d078, State: Initialized, Role: FOLLOWER
I20260929 18:35:07.093488 11955 consensus_queue.cc:260] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572967618bb4a0eb37fab57e504e8de" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 42373 } } peers { permanent_uuid: "a0da9e3be899425d8c1d8b3bab353ea9" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 32883 } } peers { permanent_uuid: "7762bdb5642b48ceb9b2e3b59c82d078" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 40297 } }
I20260929 18:35:07.094632 11955 ts_tablet_manager.cc:1428] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
W20260929 18:35:07.107170 12065 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:35:07.071823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.65:52407)
0929 18:35:07.072624 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:07.072699 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:07.074014 (+ 1315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:07.106019 (+ 32005us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":203,"threads_started":1}
W20260929 18:35:07.107824 12063 heartbeater.cc:644] Failed to heartbeat to 127.28.207.126:43295 (0 consecutive failures): Network error: Failed to ping master at 127.28.207.126:43295: Client connection negotiation failed: client connection to 127.28.207.126:43295: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 18:35:07.108839 12065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:07.108469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.65:46351)
0929 18:35:07.108672 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:07.108751 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260929 18:35:07.109783 12065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:07.109537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.65:44019)
0929 18:35:07.109675 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:07.109724 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260929 18:35:07.110113 12063 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:35:07.200155 12066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:07.200444 12066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:07.200492 12066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:07.204661 12066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:07.204741 12066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:07.204803 12066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.66
I20260929 18:35:07.210800 12066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:40155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.66:32883
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.207.66
--webserver_port=35493
--tserver_master_addrs=127.28.207.126:43295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.66
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:35:07.211982 12066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:07.213057 12066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:07.223881 12076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:07.223896 12074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:07.223905 12073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:07.223948 12066 server_base.cc:1034] running on GCE node
I20260929 18:35:07.224686 12066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:07.225409 12066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:07.226606 12066 hybrid_clock.cc:648] HybridClock initialized: now 1790706907226578 us; error 46 us; skew 500 ppm
Sep 29 18:35:07 dist-test-slave-qcpv krb5kdc[10751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706907, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:07.231762 12066 init.cc:377] Logged in from keytab as kudu/127.28.207.66@KRBTEST.COM (short username kudu)
I20260929 18:35:07.233153 12066 webserver.cc:460] Webserver started at http://127.28.207.66:35493/ using document root <none> and password file <none>
I20260929 18:35:07.234002 12066 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:07.234112 12066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:07.243047 12066 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 18:35:07.246809 12083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:07.249017 12066 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:35:07.249279 12066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/wal
uuid: "a0da9e3be899425d8c1d8b3bab353ea9"
format_stamp: "Formatted at 2026-09-29 18:35:04 on dist-test-slave-qcpv"
server_key: "b1e05f7fc82c94fd4a24e39501dc47d6"
server_key_iv: "08d7667af4b3f8e91164defd6f9d55a4"
server_key_version: "encryptionkey@0"
I20260929 18:35:07.250226 12066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:07.286908 12066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:07.291149 12066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:07.291429 12066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:07.292330 12066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:35:07.293618 12066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:35:07.293704 12066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:07.293797 12066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:35:07.293846 12066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:07.308888 12066 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.66:32883
I20260929 18:35:07.309051 12196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.66:32883 every 8 connection(s)
I20260929 18:35:07.310491 12066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
I20260929 18:35:07.311918 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 12066
I20260929 18:35:07.312448 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.67:42373
--local_ip_for_outbound_sockets=127.28.207.67
--tserver_master_addrs=127.28.207.126:43295
--webserver_port=44107
--webserver_interface=127.28.207.67
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:40155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:35:07.317101 12200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:07.314912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.66:59391)
0929 18:35:07.315748 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:07.316265 (+ 517us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":222,"threads_started":1}
W20260929 18:35:07.317821 12197 heartbeater.cc:644] Failed to heartbeat to 127.28.207.126:43295 (0 consecutive failures): Network error: Failed to ping master at 127.28.207.126:43295: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
I20260929 18:35:07.318886 12200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:07.318472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.66:39983)
0929 18:35:07.318676 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:07.318758 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 18:35:07.319837 12200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:07.319614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.66:44123)
0929 18:35:07.319743 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:07.319789 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260929 18:35:07.320084 12197 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:35:07.433636 12199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:07.433955 12199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:07.434026 12199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:07.438082 12199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:07.438174 12199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:07.438264 12199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.67
I20260929 18:35:07.443536 12199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:40155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.67:42373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.207.67
--webserver_port=44107
--tserver_master_addrs=127.28.207.126:43295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.67
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:35:07.444623 12199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:07.445629 12199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:07.456351 12208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:07.456504 12205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:07.456419 12206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:07.456483 12199 server_base.cc:1034] running on GCE node
I20260929 18:35:07.457180 12199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:07.458062 12199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:07.459301 12199 hybrid_clock.cc:648] HybridClock initialized: now 1790706907459256 us; error 54 us; skew 500 ppm
Sep 29 18:35:07 dist-test-slave-qcpv krb5kdc[10751](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706907, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:07.463037 12199 init.cc:377] Logged in from keytab as kudu/127.28.207.67@KRBTEST.COM (short username kudu)
I20260929 18:35:07.464224 12199 webserver.cc:460] Webserver started at http://127.28.207.67:44107/ using document root <none> and password file <none>
I20260929 18:35:07.465101 12199 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:07.465350 12199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:07.470515 12199 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:35:07.473448 12215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:07.475301 12199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:35:07.475488 12199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/wal
uuid: "a572967618bb4a0eb37fab57e504e8de"
format_stamp: "Formatted at 2026-09-29 18:35:04 on dist-test-slave-qcpv"
server_key: "ce4429dddf07e0305486a764b5b1e942"
server_key_iv: "35a2dc5d245391a98481adc2decf0b7b"
server_key_version: "encryptionkey@0"
I20260929 18:35:07.476204 12199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:07.490311 12199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:07.494454 12199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:07.494709 12199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:07.495615 12199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:35:07.496685 12199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:35:07.496762 12199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:07.496850 12199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:35:07.496903 12199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:07.508988 12199 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.67:42373
I20260929 18:35:07.509028 12328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.67:42373 every 8 connection(s)
I20260929 18:35:07.510200 12199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
I20260929 18:35:07.512138 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 12199
I20260929 18:35:07.515285 12332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:07.513143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.67:47597)
0929 18:35:07.513930 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:07.514398 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":268,"threads_started":1}
W20260929 18:35:07.515914 12329 heartbeater.cc:644] Failed to heartbeat to 127.28.207.126:43295 (0 consecutive failures): Network error: Failed to ping master at 127.28.207.126:43295: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
I20260929 18:35:07.516631 12332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:07.516405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.67:33249)
0929 18:35:07.516527 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:07.516579 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 18:35:07.517345 12332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:07.517152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.67:45393)
0929 18:35:07.517254 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:07.517298 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 18:35:07.517583 12329 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 18:35:08.111711 12333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:08.111110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.65:55283)
0929 18:35:08.111497 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:08.111576 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":131,"threads_started":1}
I20260929 18:35:08.321736 12334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:08.321118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.66:54299)
0929 18:35:08.321476 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:08.321582 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":85,"threads_started":1}
I20260929 18:35:08.387426 12335 raft_consensus.cc:482] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:35:08.387779 12335 raft_consensus.cc:504] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572967618bb4a0eb37fab57e504e8de" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 42373 } } peers { permanent_uuid: "a0da9e3be899425d8c1d8b3bab353ea9" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 32883 } } peers { permanent_uuid: "7762bdb5642b48ceb9b2e3b59c82d078" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 40297 } }
I20260929 18:35:08.389194 12335 leader_election.cc:288] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a572967618bb4a0eb37fab57e504e8de (127.28.207.67:42373), a0da9e3be899425d8c1d8b3bab353ea9 (127.28.207.66:32883)
Sep 29 18:35:08 dist-test-slave-qcpv krb5kdc[10751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706907, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for kudu/127.28.207.66@KRBTEST.COM
Sep 29 18:35:08 dist-test-slave-qcpv krb5kdc[10751](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706907, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for kudu/127.28.207.67@KRBTEST.COM
I20260929 18:35:08.408550 12338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:08.390100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:36535 (local address 127.28.207.66:32883)
0929 18:35:08.390548 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:08.390572 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:08.390614 (+ 42us) server_negotiation.cc:407] Connection header received
0929 18:35:08.391465 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:08.391472 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:08.391870 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:08.392217 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:08.393546 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:08.395282 (+ 1736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:08.397461 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:08.398326 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:08.398526 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:08.403125 (+ 4599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:08.403131 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:08.403144 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:08.403179 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:08.405632 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:08.406335 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:08.406341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:08.406345 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:08.406484 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:08.406848 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:08.406853 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:08.406894 (+ 41us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:08.407362 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:08.407510 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:08.408079 (+ 569us) server_negotiation.cc:299] Negotiation successful
0929 18:35:08.408356 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":151,"threads_started":1}
I20260929 18:35:08.408540 12336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:08.389636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:32883 (local address 127.28.207.65:36535)
0929 18:35:08.390159 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:08.390178 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:08.390344 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:08.392149 (+ 1805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:08.392152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:08.392803 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:08.393342 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:08.393359 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:08.395494 (+ 2135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:08.395498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:08.397267 (+ 1769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:08.397282 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:08.398548 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:08.398552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:08.398925 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:08.399955 (+ 1030us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:08.400026 (+ 71us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:08.402783 (+ 2757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:08.405790 (+ 3007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:08.405797 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:08.405813 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:08.406195 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:08.406588 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:08.406594 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:08.406596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:08.406748 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:08.407480 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:08.407487 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:08.407767 (+ 280us) client_negotiation.cc:769] Sending connection context
0929 18:35:08.407958 (+ 191us) client_negotiation.cc:240] Negotiation successful
0929 18:35:08.408338 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":342,"spinlock_wait_cycles":17920,"thread_start_us":103,"threads_started":1}
I20260929 18:35:08.408766 12333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:08.389636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:42373 (local address 127.28.207.65:35689)
0929 18:35:08.389917 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:08.389935 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:08.390065 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:08.392101 (+ 2036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:08.392107 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:08.392804 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:08.393341 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:08.393358 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:08.395494 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:08.395498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:08.397267 (+ 1769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:08.397282 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:08.398548 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:08.398560 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:08.398912 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:08.399955 (+ 1043us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:08.400026 (+ 71us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:08.403180 (+ 3154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:08.405992 (+ 2812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:08.405996 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:08.405998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:08.406271 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:08.406923 (+ 652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:08.406927 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:08.406930 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:08.407040 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:08.407874 (+ 834us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:08.407878 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:08.408013 (+ 135us) client_negotiation.cc:769] Sending connection context
0929 18:35:08.408143 (+ 130us) client_negotiation.cc:240] Negotiation successful
0929 18:35:08.408467 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":412,"spinlock_wait_cycles":104704}
I20260929 18:35:08.409684 12337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:08.389951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:35689 (local address 127.28.207.67:42373)
0929 18:35:08.390388 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:08.390408 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:08.390462 (+ 54us) server_negotiation.cc:407] Connection header received
0929 18:35:08.391265 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:08.391272 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:08.391674 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:08.392001 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:08.393601 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:08.395283 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:08.397460 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:08.398326 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:08.398526 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:08.403487 (+ 4961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:08.403494 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:08.403512 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:08.403553 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:08.405825 (+ 2272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:08.406470 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:08.406477 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:08.406546 (+ 69us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:08.406727 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:08.407153 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:08.407158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:08.407160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:08.407730 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:08.407893 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:08.408369 (+ 476us) server_negotiation.cc:299] Negotiation successful
0929 18:35:08.409524 (+ 1155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":131,"threads_started":1}
W20260929 18:35:08.411134 11949 leader_election.cc:341] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0da9e3be899425d8c1d8b3bab353ea9 (127.28.207.66:32883): Not found: Tablet not found: fac673c0b4db4b8bb151f0bf9477d07b
W20260929 18:35:08.412137 11951 leader_election.cc:341] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a572967618bb4a0eb37fab57e504e8de (127.28.207.67:42373): Not found: Tablet not found: fac673c0b4db4b8bb151f0bf9477d07b
I20260929 18:35:08.412222 11951 leader_election.cc:302] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7762bdb5642b48ceb9b2e3b59c82d078; no voters: a0da9e3be899425d8c1d8b3bab353ea9, a572967618bb4a0eb37fab57e504e8de
I20260929 18:35:08.412612 12335 raft_consensus.cc:2726] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:35:08.518767 12339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:08.518292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.67:41037)
0929 18:35:08.518597 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:08.518661 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":96,"threads_started":1}
I20260929 18:35:09.113530 12340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:09.112913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.65:60171)
0929 18:35:09.113331 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:09.113421 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":106,"threads_started":1}
I20260929 18:35:09.323546 12341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:09.322839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.66:42921)
0929 18:35:09.323250 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:09.323342 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":99,"threads_started":1}
I20260929 18:35:09.520466 12342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:09.519670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.67:34619)
0929 18:35:09.520049 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:09.520129 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":114,"threads_started":1}
I20260929 18:35:10.115453 12343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:10.114780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.65:51965)
0929 18:35:10.115257 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:10.115351 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":180,"threads_started":1}
I20260929 18:35:10.325322 12344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:10.324627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.66:52653)
0929 18:35:10.325060 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:10.325161 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":81,"threads_started":1}
I20260929 18:35:10.394207 12345 raft_consensus.cc:482] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:35:10.394364 12345 raft_consensus.cc:504] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572967618bb4a0eb37fab57e504e8de" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 42373 } } peers { permanent_uuid: "a0da9e3be899425d8c1d8b3bab353ea9" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 32883 } } peers { permanent_uuid: "7762bdb5642b48ceb9b2e3b59c82d078" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 40297 } }
I20260929 18:35:10.394891 12345 leader_election.cc:288] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a572967618bb4a0eb37fab57e504e8de (127.28.207.67:42373), a0da9e3be899425d8c1d8b3bab353ea9 (127.28.207.66:32883)
W20260929 18:35:10.395975 11949 leader_election.cc:341] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0da9e3be899425d8c1d8b3bab353ea9 (127.28.207.66:32883): Not found: Tablet not found: fac673c0b4db4b8bb151f0bf9477d07b
W20260929 18:35:10.396260 11951 leader_election.cc:341] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a572967618bb4a0eb37fab57e504e8de (127.28.207.67:42373): Not found: Tablet not found: fac673c0b4db4b8bb151f0bf9477d07b
I20260929 18:35:10.396356 11951 leader_election.cc:302] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7762bdb5642b48ceb9b2e3b59c82d078; no voters: a0da9e3be899425d8c1d8b3bab353ea9, a572967618bb4a0eb37fab57e504e8de
I20260929 18:35:10.396498 12345 raft_consensus.cc:2726] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:35:10.522094 12346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:10.521506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.67:54347)
0929 18:35:10.521870 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:10.521959 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":87,"threads_started":1}
I20260929 18:35:11.117332 12347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:11.116767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.65:42279)
0929 18:35:11.117138 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:11.117224 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":104,"threads_started":1}
I20260929 18:35:11.326974 12348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:11.326412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.66:32953)
0929 18:35:11.326729 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:11.326796 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":97,"threads_started":1}
I20260929 18:35:11.523921 12349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:11.523301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.67:47545)
0929 18:35:11.523681 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:11.523770 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":106,"threads_started":1}
I20260929 18:35:11.909587 12350 raft_consensus.cc:482] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:35:11.909782 12350 raft_consensus.cc:504] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572967618bb4a0eb37fab57e504e8de" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 42373 } } peers { permanent_uuid: "a0da9e3be899425d8c1d8b3bab353ea9" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 32883 } } peers { permanent_uuid: "7762bdb5642b48ceb9b2e3b59c82d078" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 40297 } }
I20260929 18:35:11.910460 12350 leader_election.cc:288] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a572967618bb4a0eb37fab57e504e8de (127.28.207.67:42373), a0da9e3be899425d8c1d8b3bab353ea9 (127.28.207.66:32883)
W20260929 18:35:11.911629 11951 leader_election.cc:341] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a572967618bb4a0eb37fab57e504e8de (127.28.207.67:42373): Not found: Tablet not found: fac673c0b4db4b8bb151f0bf9477d07b
W20260929 18:35:11.911813 11949 leader_election.cc:341] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0da9e3be899425d8c1d8b3bab353ea9 (127.28.207.66:32883): Not found: Tablet not found: fac673c0b4db4b8bb151f0bf9477d07b
I20260929 18:35:11.911985 11949 leader_election.cc:302] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7762bdb5642b48ceb9b2e3b59c82d078; no voters: a0da9e3be899425d8c1d8b3bab353ea9, a572967618bb4a0eb37fab57e504e8de
I20260929 18:35:11.912186 12350 raft_consensus.cc:2726] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:35:12.119426 12351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:12.118701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.65:56935)
0929 18:35:12.119205 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:12.119295 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":180,"threads_started":1}
I20260929 18:35:12.328977 12352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:12.328282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.66:43455)
0929 18:35:12.328758 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:12.328836 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":133,"threads_started":1}
I20260929 18:35:12.525765 12353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:12.525169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.67:36609)
0929 18:35:12.525568 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:12.525649 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":122,"threads_started":1}
I20260929 18:35:13.121098 12354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:13.120520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.65:37649)
0929 18:35:13.120905 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:13.120990 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":107,"threads_started":1}
I20260929 18:35:13.330535 12355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:13.330063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.66:35909)
0929 18:35:13.330373 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:13.330433 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":85,"threads_started":1}
I20260929 18:35:13.527607 12356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:13.526756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.67:32795)
0929 18:35:13.527194 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:13.527380 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":104,"threads_started":1}
I20260929 18:35:14.122860 12357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:14.122114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.65:40597)
0929 18:35:14.122562 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:14.122665 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":90,"threads_started":1}
I20260929 18:35:14.332163 12358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:14.331645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.66:42477)
0929 18:35:14.331972 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:14.332050 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":125,"threads_started":1}
I20260929 18:35:14.529448 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:14.528808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.67:56249)
0929 18:35:14.529223 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:14.529312 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":89,"threads_started":1}
I20260929 18:35:15.124871 12360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:15.124176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.65:54703)
0929 18:35:15.124637 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:15.124735 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":92,"threads_started":1}
I20260929 18:35:15.333837 12361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:15.333204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.66:35931)
0929 18:35:15.333593 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:15.333699 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":81,"threads_started":1}
I20260929 18:35:15.531390 12362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:15.530808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.67:46269)
0929 18:35:15.531215 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:15.531283 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":90,"threads_started":1}
I20260929 18:35:15.577550 12363 raft_consensus.cc:482] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:35:15.577816 12363 raft_consensus.cc:504] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572967618bb4a0eb37fab57e504e8de" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 42373 } } peers { permanent_uuid: "a0da9e3be899425d8c1d8b3bab353ea9" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 32883 } } peers { permanent_uuid: "7762bdb5642b48ceb9b2e3b59c82d078" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 40297 } }
I20260929 18:35:15.578368 12363 leader_election.cc:288] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a572967618bb4a0eb37fab57e504e8de (127.28.207.67:42373), a0da9e3be899425d8c1d8b3bab353ea9 (127.28.207.66:32883)
W20260929 18:35:15.579416 11951 leader_election.cc:341] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a572967618bb4a0eb37fab57e504e8de (127.28.207.67:42373): Not found: Tablet not found: fac673c0b4db4b8bb151f0bf9477d07b
W20260929 18:35:15.579566 11949 leader_election.cc:341] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0da9e3be899425d8c1d8b3bab353ea9 (127.28.207.66:32883): Not found: Tablet not found: fac673c0b4db4b8bb151f0bf9477d07b
I20260929 18:35:15.579713 11949 leader_election.cc:302] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7762bdb5642b48ceb9b2e3b59c82d078; no voters: a0da9e3be899425d8c1d8b3bab353ea9, a572967618bb4a0eb37fab57e504e8de
I20260929 18:35:15.579959 12363 raft_consensus.cc:2726] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:35:16.126924 12364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:16.126204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.65:35745)
0929 18:35:16.126672 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:16.126750 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":156,"threads_started":1}
I20260929 18:35:16.336076 12365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:16.335480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.66:54317)
0929 18:35:16.335855 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:16.335953 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":104,"threads_started":1}
I20260929 18:35:16.533383 12366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:16.532690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.67:39667)
0929 18:35:16.533178 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:16.533264 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":144,"threads_started":1}
I20260929 18:35:17.128878 12367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:17.128200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.65:42795)
0929 18:35:17.128580 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:17.128694 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":110,"threads_started":1}
I20260929 18:35:17.337870 12368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:17.337263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.66:49301)
0929 18:35:17.337632 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:17.337711 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":162,"threads_started":1}
I20260929 18:35:17.535284 12369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:17.534672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.67:35747)
0929 18:35:17.535097 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:17.535174 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":106,"threads_started":1}
I20260929 18:35:18.130630 12370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:18.130031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.65:46365)
0929 18:35:18.130410 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:18.130518 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":123,"threads_started":1}
I20260929 18:35:18.339598 12371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:18.339057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.66:38429)
0929 18:35:18.339399 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:18.339467 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":101,"threads_started":1}
I20260929 18:35:18.537134 12372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:18.536380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.67:51183)
0929 18:35:18.536891 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:18.536996 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":102,"threads_started":1}
I20260929 18:35:19.132496 12373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:19.131848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.65:34929)
0929 18:35:19.132269 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:19.132378 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":110,"threads_started":1}
I20260929 18:35:19.341719 12374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:19.340626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.66:34531)
0929 18:35:19.341454 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:19.341575 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":101,"threads_started":1}
I20260929 18:35:19.538594 12375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:19.538111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.67:50515)
0929 18:35:19.538420 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:19.538487 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":106,"threads_started":1}
I20260929 18:35:20.134197 12376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:20.133460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.65:50427)
0929 18:35:20.133893 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:20.134012 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":82,"threads_started":1}
I20260929 18:35:20.343833 12377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:20.343198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.66:35081)
0929 18:35:20.343638 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:20.343703 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":106,"threads_started":1}
I20260929 18:35:20.540385 12378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:20.539823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.67:56191)
0929 18:35:20.540200 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:20.540277 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":103,"threads_started":1}
I20260929 18:35:20.810654 12379 raft_consensus.cc:482] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:35:20.810823 12379 raft_consensus.cc:504] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a572967618bb4a0eb37fab57e504e8de" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 42373 } } peers { permanent_uuid: "a0da9e3be899425d8c1d8b3bab353ea9" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 32883 } } peers { permanent_uuid: "7762bdb5642b48ceb9b2e3b59c82d078" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 40297 } }
I20260929 18:35:20.811474 12379 leader_election.cc:288] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a572967618bb4a0eb37fab57e504e8de (127.28.207.67:42373), a0da9e3be899425d8c1d8b3bab353ea9 (127.28.207.66:32883)
W20260929 18:35:20.812328 11951 leader_election.cc:341] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a572967618bb4a0eb37fab57e504e8de (127.28.207.67:42373): Not found: Tablet not found: fac673c0b4db4b8bb151f0bf9477d07b
W20260929 18:35:20.812701 11949 leader_election.cc:341] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a0da9e3be899425d8c1d8b3bab353ea9 (127.28.207.66:32883): Not found: Tablet not found: fac673c0b4db4b8bb151f0bf9477d07b
I20260929 18:35:20.812844 11949 leader_election.cc:302] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7762bdb5642b48ceb9b2e3b59c82d078; no voters: a0da9e3be899425d8c1d8b3bab353ea9, a572967618bb4a0eb37fab57e504e8de
I20260929 18:35:20.813082 12379 raft_consensus.cc:2726] T fac673c0b4db4b8bb151f0bf9477d07b P 7762bdb5642b48ceb9b2e3b59c82d078 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 18:35:21.136040 12380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:21.135405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.65:57489)
0929 18:35:21.135824 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:21.135920 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":106,"threads_started":1}
I20260929 18:35:21.345889 12381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:21.345070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.66:45559)
0929 18:35:21.345531 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:21.345726 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":148,"threads_started":1}
I20260929 18:35:21.543669 12382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:21.542583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.67:60917)
0929 18:35:21.543302 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:21.543393 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":159,"threads_started":1}
I20260929 18:35:22.138162 12383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:22.137405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.65:42337)
0929 18:35:22.137867 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:22.137996 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":136,"threads_started":1}
I20260929 18:35:22.348189 12384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:22.347745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43295 (local address 127.28.207.66:37411)
0929 18:35:22.348041 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:22.348090 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:43295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":96,"threads_started":1}
I20260929 18:35:22.516932 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 11899
I20260929 18:35:22.523849 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 12066
I20260929 18:35:22.530766 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 12199
2026-09-29T18:35:22Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19723 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:35:22 dist-test-slave-qcpv krb5kdc[12387](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:35:22 dist-test-slave-qcpv krb5kdc[12387](info): set up 2 sockets
Sep 29 18:35:22 dist-test-slave-qcpv krb5kdc[12387](info): commencing operation
krb5kdc: starting...
W20260929 18:35:23.595301 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:35:23 dist-test-slave-qcpv krb5kdc[12387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:35:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:35:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.207.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
I20260929 18:35:23.742240 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:36041
--webserver_interface=127.28.207.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:42001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:36041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:23.883375 12400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:23.883832 12400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:23.883917 12400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:23.888429 12400 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 18:35:23.888550 12400 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 18:35:23.888578 12400 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 18:35:23.888602 12400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:35:23.888646 12400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:23.888712 12400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:23.888779 12400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:35:23.888821 12400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:35:23.895977 12400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:42001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:36041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:36041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:35:23.897542 12400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:23.899056 12400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:23.910485 12406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:23.910530 12408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:23.910511 12400 server_base.cc:1034] running on GCE node
W20260929 18:35:23.910894 12405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:23.911708 12400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:23.913462 12400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:23.914675 12400 hybrid_clock.cc:648] HybridClock initialized: now 1790706923914627 us; error 72 us; skew 500 ppm
Sep 29 18:35:23 dist-test-slave-qcpv krb5kdc[12387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706923, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:23.918931 12400 init.cc:377] Logged in from keytab as kudu/127.28.207.126@KRBTEST.COM (short username kudu)
I20260929 18:35:23.920174 12400 webserver.cc:460] Webserver started at http://127.28.207.126:36549/ using document root <none> and password file <none>
I20260929 18:35:23.920892 12400 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:23.920990 12400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:23.921288 12400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:23.924063 12400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "30e2464e79e64d83a9d4b044ec3e3a46"
format_stamp: "Formatted at 2026-09-29 18:35:23 on dist-test-slave-qcpv"
server_key: "42857d24f67d8174476fabdc8ede8779"
server_key_iv: "29b71d3de15d499b25bb7a1c75d33bd4"
server_key_version: "encryptionkey@0"
I20260929 18:35:23.924767 12400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "30e2464e79e64d83a9d4b044ec3e3a46"
format_stamp: "Formatted at 2026-09-29 18:35:23 on dist-test-slave-qcpv"
server_key: "42857d24f67d8174476fabdc8ede8779"
server_key_iv: "29b71d3de15d499b25bb7a1c75d33bd4"
server_key_version: "encryptionkey@0"
I20260929 18:35:23.929669 12400 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 18:35:23.933012 12415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:23.934791 12400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:35:23.935040 12400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "30e2464e79e64d83a9d4b044ec3e3a46"
format_stamp: "Formatted at 2026-09-29 18:35:23 on dist-test-slave-qcpv"
server_key: "42857d24f67d8174476fabdc8ede8779"
server_key_iv: "29b71d3de15d499b25bb7a1c75d33bd4"
server_key_version: "encryptionkey@0"
I20260929 18:35:23.935199 12400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:23.963214 12400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:23.967545 12400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:23.967829 12400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:23.978338 12400 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:36041
I20260929 18:35:23.978675 12467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:36041 every 8 connection(s)
I20260929 18:35:23.979715 12400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:35:23.981305 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 12400
I20260929 18:35:23.981446 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:35:23.981818 29501 external_mini_cluster.cc:1428] Setting key 68af570edc57ab5e6d4581f6a4f4ad53
I20260929 18:35:23.983723 12468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:23.991107 12468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46: Bootstrap starting.
Sep 29 18:35:23 dist-test-slave-qcpv krb5kdc[12387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:35:23.993971 12468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:23.995045 12468 log.cc:824] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:23.997884 12468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46: No bootstrap required, opened a new log
I20260929 18:35:23.999817 12471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:23.983636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38686 (local address 127.28.207.126:36041)
0929 18:35:23.984187 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:23.984206 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:23.984261 (+ 55us) server_negotiation.cc:407] Connection header received
0929 18:35:23.985068 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:23.985088 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:23.985502 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:23.985855 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:23.986938 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:23.988119 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:23.989387 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:23.989961 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:23.990114 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:23.993622 (+ 3508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:23.993630 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:23.993650 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:23.993725 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:23.995955 (+ 2230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:23.996579 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:23.996585 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:23.996592 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:23.996768 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:23.997126 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:23.997129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:23.997131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:23.997696 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:23.997894 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:23.998265 (+ 371us) server_negotiation.cc:299] Negotiation successful
0929 18:35:23.998601 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":210,"threads_started":1}
I20260929 18:35:24.003113 12468 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e2464e79e64d83a9d4b044ec3e3a46" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 36041 } }
I20260929 18:35:24.003484 12468 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:24.003608 12468 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30e2464e79e64d83a9d4b044ec3e3a46, State: Initialized, Role: FOLLOWER
I20260929 18:35:24.004122 12468 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e2464e79e64d83a9d4b044ec3e3a46" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 36041 } }
I20260929 18:35:24.004271 12468 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:35:24.004370 12468 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:35:24.004478 12468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:24.005604 12468 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e2464e79e64d83a9d4b044ec3e3a46" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 36041 } }
I20260929 18:35:24.006011 12468 leader_election.cc:302] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30e2464e79e64d83a9d4b044ec3e3a46; no voters:
I20260929 18:35:24.006397 12468 leader_election.cc:288] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:35:24.006649 12473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:24.007097 12473 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [term 1 LEADER]: Becoming Leader. State: Replica: 30e2464e79e64d83a9d4b044ec3e3a46, State: Running, Role: LEADER
I20260929 18:35:24.007530 12473 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e2464e79e64d83a9d4b044ec3e3a46" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 36041 } }
I20260929 18:35:24.008050 12468 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:35:24.009600 12475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30e2464e79e64d83a9d4b044ec3e3a46. Latest consensus state: current_term: 1 leader_uuid: "30e2464e79e64d83a9d4b044ec3e3a46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e2464e79e64d83a9d4b044ec3e3a46" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 36041 } } }
I20260929 18:35:24.009780 12475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:24.010120 12474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30e2464e79e64d83a9d4b044ec3e3a46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e2464e79e64d83a9d4b044ec3e3a46" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 36041 } } }
I20260929 18:35:24.010228 12474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [sys.catalog]: This master's current role is: LEADER
W20260929 18:35:24.014358 12488 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:35:24.014467 12488 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:35:24.014573 12482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:35:24.015545 12482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:35:24.024192 12482 catalog_manager.cc:1347] Generated new cluster ID: b8357524696649b287da1fcb2b016672
I20260929 18:35:24.024355 12482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:35:24.055178 12482 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:35:24.056293 12482 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:35:24.069947 12482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46: Generated new TSK 0
I20260929 18:35:24.070808 12482 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.207.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab.
Entry for principal kudu/127.28.207.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab.
I20260929 18:35:24.117843 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.65:0
--local_ip_for_outbound_sockets=127.28.207.65
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:36041
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:42001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:24.251010 12494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:24.251374 12494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:24.251454 12494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:24.255885 12494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:24.255976 12494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:24.256090 12494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.65
I20260929 18:35:24.261843 12494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:42001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:36041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.65
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:35:24.263134 12494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:24.264192 12494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:24.274941 12500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:24.274952 12499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:24.274988 12502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:24.275460 12494 server_base.cc:1034] running on GCE node
I20260929 18:35:24.275954 12494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:24.276636 12494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:24.277848 12494 hybrid_clock.cc:648] HybridClock initialized: now 1790706924277817 us; error 87 us; skew 500 ppm
Sep 29 18:35:24 dist-test-slave-qcpv krb5kdc[12387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706924, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:24.281070 12494 init.cc:377] Logged in from keytab as kudu/127.28.207.65@KRBTEST.COM (short username kudu)
I20260929 18:35:24.282230 12494 webserver.cc:460] Webserver started at http://127.28.207.65:34605/ using document root <none> and password file <none>
I20260929 18:35:24.282833 12494 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:24.282981 12494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:24.283223 12494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:24.285422 12494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/data/instance:
uuid: "9dfe7e3bf3674eed89c366f59653a594"
format_stamp: "Formatted at 2026-09-29 18:35:24 on dist-test-slave-qcpv"
server_key: "344cfc47f7a3d5e65f98bd25d917a8c4"
server_key_iv: "ba61fbd4b04313419210b3d5a5f13ac9"
server_key_version: "encryptionkey@0"
I20260929 18:35:24.286002 12494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance:
uuid: "9dfe7e3bf3674eed89c366f59653a594"
format_stamp: "Formatted at 2026-09-29 18:35:24 on dist-test-slave-qcpv"
server_key: "344cfc47f7a3d5e65f98bd25d917a8c4"
server_key_iv: "ba61fbd4b04313419210b3d5a5f13ac9"
server_key_version: "encryptionkey@0"
I20260929 18:35:24.290402 12494 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:35:24.293339 12509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:24.295064 12494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:35:24.295217 12494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/wal
uuid: "9dfe7e3bf3674eed89c366f59653a594"
format_stamp: "Formatted at 2026-09-29 18:35:24 on dist-test-slave-qcpv"
server_key: "344cfc47f7a3d5e65f98bd25d917a8c4"
server_key_iv: "ba61fbd4b04313419210b3d5a5f13ac9"
server_key_version: "encryptionkey@0"
I20260929 18:35:24.295338 12494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:24.309317 12494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:24.313284 12494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:24.313474 12494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:24.314139 12494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:35:24.315130 12494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:35:24.315181 12494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:24.315224 12494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:35:24.315253 12494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:24.326718 12494 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.65:46723
I20260929 18:35:24.326781 12622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.65:46723 every 8 connection(s)
I20260929 18:35:24.327890 12494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
I20260929 18:35:24.337423 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 12494
I20260929 18:35:24.337589 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance
I20260929 18:35:24.337950 29501 external_mini_cluster.cc:1428] Setting key 1e66d66ddd89ffcc75b2970ff33d82ee
Sep 29 18:35:24 dist-test-slave-qcpv krb5kdc[12387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706924, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:35:24.345777 12471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:24.330103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:46329 (local address 127.28.207.126:36041)
0929 18:35:24.330308 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:24.330313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:24.331714 (+ 1401us) server_negotiation.cc:407] Connection header received
0929 18:35:24.332787 (+ 1073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:24.332790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:24.332858 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:24.332988 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:24.335214 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:24.335801 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:24.337213 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:24.337613 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:24.337827 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:24.341222 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:24.341229 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:24.341235 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:24.341273 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:24.343098 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:24.343848 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:24.343850 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:24.343852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:24.343932 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:24.344382 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:24.344385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:24.344386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:24.344585 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:24.344673 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:24.345511 (+ 838us) server_negotiation.cc:299] Negotiation successful
0929 18:35:24.345646 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 18:35:24.347458 12625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:24.330528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.65:46329)
0929 18:35:24.331510 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:24.331563 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:24.332514 (+ 951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:24.333640 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:24.333650 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:24.334127 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:24.335016 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:24.335035 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:24.336003 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:24.336006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:24.337045 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:24.337055 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:24.337882 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:24.337885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:24.338134 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:24.338946 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:24.338973 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:24.341001 (+ 2028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:24.343238 (+ 2237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:24.343249 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:24.343275 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:24.343721 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:24.344035 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:24.344041 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:24.344046 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:24.344260 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:24.344710 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:24.344721 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:24.345171 (+ 450us) client_negotiation.cc:769] Sending connection context
0929 18:35:24.345443 (+ 272us) client_negotiation.cc:240] Negotiation successful
0929 18:35:24.345918 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"thread_start_us":150,"threads_started":1}
I20260929 18:35:24.348963 12623 heartbeater.cc:347] Connected to a master server at 127.28.207.126:36041
I20260929 18:35:24.349337 12623 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.28.207.66@KRBTEST.COM; defaulting to no policy
I20260929 18:35:24.350041 12623 heartbeater.cc:511] Master 127.28.207.126:36041 requested a full tablet report, sending...
I20260929 18:35:24.351727 12432 ts_manager.cc:194] Registered new tserver with Master: 9dfe7e3bf3674eed89c366f59653a594 (127.28.207.65:46723)
I20260929 18:35:24.352994 12432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.65@KRBTEST.COM'} at 127.28.207.65:46329
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab.
Entry for principal kudu/127.28.207.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab.
I20260929 18:35:24.375186 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.66:0
--local_ip_for_outbound_sockets=127.28.207.66
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:36041
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:42001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:24.514994 12628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:24.515353 12628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:24.515509 12628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:24.519917 12628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:24.520028 12628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:24.520125 12628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.66
I20260929 18:35:24.526350 12628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:42001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:36041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.66
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:35:24.527940 12628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:24.529402 12628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:24.541950 12633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:24.542181 12636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:24.542141 12634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:24.542382 12628 server_base.cc:1034] running on GCE node
I20260929 18:35:24.543239 12628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:24.544489 12628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:24.545725 12628 hybrid_clock.cc:648] HybridClock initialized: now 1790706924545646 us; error 92 us; skew 500 ppm
Sep 29 18:35:24 dist-test-slave-qcpv krb5kdc[12387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706924, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:24.549434 12628 init.cc:377] Logged in from keytab as kudu/127.28.207.66@KRBTEST.COM (short username kudu)
I20260929 18:35:24.550635 12628 webserver.cc:460] Webserver started at http://127.28.207.66:44385/ using document root <none> and password file <none>
I20260929 18:35:24.551420 12628 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:24.551493 12628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:24.551704 12628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:24.555462 12628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/data/instance:
uuid: "12e32b320f0349d18c5c4811cae01a0d"
format_stamp: "Formatted at 2026-09-29 18:35:24 on dist-test-slave-qcpv"
server_key: "501e0055f9101ec05c9cb1cde80eca59"
server_key_iv: "2c9fd2e5322082e5eaef49d2d004141f"
server_key_version: "encryptionkey@0"
I20260929 18:35:24.556175 12628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance:
uuid: "12e32b320f0349d18c5c4811cae01a0d"
format_stamp: "Formatted at 2026-09-29 18:35:24 on dist-test-slave-qcpv"
server_key: "501e0055f9101ec05c9cb1cde80eca59"
server_key_iv: "2c9fd2e5322082e5eaef49d2d004141f"
server_key_version: "encryptionkey@0"
I20260929 18:35:24.561527 12628 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 18:35:24.564668 12643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:24.566766 12628 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260929 18:35:24.567062 12628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/wal
uuid: "12e32b320f0349d18c5c4811cae01a0d"
format_stamp: "Formatted at 2026-09-29 18:35:24 on dist-test-slave-qcpv"
server_key: "501e0055f9101ec05c9cb1cde80eca59"
server_key_iv: "2c9fd2e5322082e5eaef49d2d004141f"
server_key_version: "encryptionkey@0"
I20260929 18:35:24.567240 12628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:24.590583 12628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:24.595367 12628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:24.595633 12628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:24.596433 12628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:35:24.597674 12628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:35:24.597756 12628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:24.597841 12628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:35:24.597884 12628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:24.612401 12628 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.66:46547
I20260929 18:35:24.612471 12756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.66:46547 every 8 connection(s)
I20260929 18:35:24.614014 12628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
I20260929 18:35:24.615166 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 12628
I20260929 18:35:24.615346 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance
I20260929 18:35:24.615845 29501 external_mini_cluster.cc:1428] Setting key 7a342a7fd33a34ea76b69be7c224e073
Sep 29 18:35:24 dist-test-slave-qcpv krb5kdc[12387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706924, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.207.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.67@KRBTEST.COM" created.
I20260929 18:35:24.633549 12471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:24.616950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:42533 (local address 127.28.207.126:36041)
0929 18:35:24.617096 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:24.617099 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:24.618792 (+ 1693us) server_negotiation.cc:407] Connection header received
0929 18:35:24.620126 (+ 1334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:24.620130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:24.620190 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:24.620263 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:24.621771 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:24.622303 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:24.623689 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:24.623923 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:24.624036 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:24.627632 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:24.627636 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:24.627639 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:24.627671 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:24.629368 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:24.630558 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:24.630561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:24.630562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:24.630656 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:24.631370 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:24.631374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:24.631375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:24.631761 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:24.631881 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:24.632932 (+ 1051us) server_negotiation.cc:299] Negotiation successful
0929 18:35:24.633287 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:35:24.634156 12759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:24.617321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.66:42533)
0929 18:35:24.618603 (+ 1282us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:24.618662 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:24.619864 (+ 1202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:24.620451 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:24.620459 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:24.620985 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:24.621583 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:24.621598 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:24.622437 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:24.622440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:24.623542 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:24.623553 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:24.624109 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:24.624114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:24.624300 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:24.624977 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:24.624994 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:24.627262 (+ 2268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:24.629568 (+ 2306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:24.629576 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:24.629660 (+ 84us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:24.630350 (+ 690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:24.630804 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:24.630808 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:24.630811 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:24.631141 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:24.631916 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:24.631928 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:24.632352 (+ 424us) client_negotiation.cc:769] Sending connection context
0929 18:35:24.632600 (+ 248us) client_negotiation.cc:240] Negotiation successful
0929 18:35:24.633019 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1008,"thread_start_us":750,"threads_started":1}
I20260929 18:35:24.635751 12757 heartbeater.cc:347] Connected to a master server at 127.28.207.126:36041
I20260929 18:35:24.636379 12757 heartbeater.cc:464] Registering TS with master...
I20260929 18:35:24.637238 12757 heartbeater.cc:511] Master 127.28.207.126:36041 requested a full tablet report, sending...
I20260929 18:35:24.638918 12432 ts_manager.cc:194] Registered new tserver with Master: 12e32b320f0349d18c5c4811cae01a0d (127.28.207.66:46547)
I20260929 18:35:24.639901 12432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.66@KRBTEST.COM'} at 127.28.207.66:42533
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab.
Entry for principal kudu/127.28.207.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab.
I20260929 18:35:24.654368 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.67:0
--local_ip_for_outbound_sockets=127.28.207.67
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:36041
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:42001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:24.795485 12762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:24.795814 12762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:24.795878 12762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:24.800222 12762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:24.800307 12762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:24.800372 12762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.67
I20260929 18:35:24.806604 12762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:42001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:36041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.67
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:35:24.807945 12762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:24.809158 12762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:24.820811 12768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:24.821033 12762 server_base.cc:1034] running on GCE node
W20260929 18:35:24.820874 12767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:24.821408 12770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:24.821964 12762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:24.822672 12762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:24.826058 12762 hybrid_clock.cc:648] HybridClock initialized: now 1790706924826011 us; error 51 us; skew 500 ppm
Sep 29 18:35:24 dist-test-slave-qcpv krb5kdc[12387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706924, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:24.835479 12762 init.cc:377] Logged in from keytab as kudu/127.28.207.67@KRBTEST.COM (short username kudu)
I20260929 18:35:24.836696 12762 webserver.cc:460] Webserver started at http://127.28.207.67:36799/ using document root <none> and password file <none>
I20260929 18:35:24.837317 12762 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:24.837386 12762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:24.837576 12762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:35:24.839875 12762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/data/instance:
uuid: "74087a8bd66b403d8c9a22b68b090fd7"
format_stamp: "Formatted at 2026-09-29 18:35:24 on dist-test-slave-qcpv"
server_key: "50f25d43b0513d5e9699c43900c84c62"
server_key_iv: "6ec32ed61e9960a8905b66409d1baf67"
server_key_version: "encryptionkey@0"
I20260929 18:35:24.840490 12762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance:
uuid: "74087a8bd66b403d8c9a22b68b090fd7"
format_stamp: "Formatted at 2026-09-29 18:35:24 on dist-test-slave-qcpv"
server_key: "50f25d43b0513d5e9699c43900c84c62"
server_key_iv: "6ec32ed61e9960a8905b66409d1baf67"
server_key_version: "encryptionkey@0"
I20260929 18:35:24.845316 12762 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 18:35:24.849014 12777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:24.851357 12762 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:35:24.851495 12762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/wal
uuid: "74087a8bd66b403d8c9a22b68b090fd7"
format_stamp: "Formatted at 2026-09-29 18:35:24 on dist-test-slave-qcpv"
server_key: "50f25d43b0513d5e9699c43900c84c62"
server_key_iv: "6ec32ed61e9960a8905b66409d1baf67"
server_key_version: "encryptionkey@0"
I20260929 18:35:24.851820 12762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:24.868709 12762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:24.874138 12762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:24.874428 12762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:24.875284 12762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:35:24.876384 12762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:35:24.876471 12762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:24.876552 12762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:35:24.876596 12762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:24.888309 12762 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.67:35193
I20260929 18:35:24.888332 12890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.67:35193 every 8 connection(s)
I20260929 18:35:24.889444 12762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
I20260929 18:35:24.894258 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 12762
I20260929 18:35:24.894423 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance
I20260929 18:35:24.894933 29501 external_mini_cluster.cc:1428] Setting key 7ad877699a7b1774bcb3ee132ae26648
Sep 29 18:35:24 dist-test-slave-qcpv krb5kdc[12387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706924, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:35:24.907207 12471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:24.891971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:54811 (local address 127.28.207.126:36041)
0929 18:35:24.892204 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:24.892209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:24.893427 (+ 1218us) server_negotiation.cc:407] Connection header received
0929 18:35:24.895153 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:24.895156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:24.895219 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:24.895332 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:24.897141 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:24.897919 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:24.899441 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:24.899757 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:24.899857 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:24.902957 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:24.902960 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:24.902962 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:24.902991 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:24.904622 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:24.905311 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:24.905314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:24.905315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:24.905401 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:24.905896 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:24.905899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:24.905900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:24.906096 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:24.906186 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:24.906853 (+ 667us) server_negotiation.cc:299] Negotiation successful
0929 18:35:24.907049 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260929 18:35:24.908643 12893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:24.892332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.67:54811)
0929 18:35:24.893216 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:24.893271 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:24.894806 (+ 1535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:24.895566 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:24.895575 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:24.896142 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:24.896974 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:24.896992 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:24.898078 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:24.898081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:24.899304 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:24.899315 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:24.899969 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:24.899977 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:24.900178 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:24.900874 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:24.900892 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:24.902690 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:24.904797 (+ 2107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:24.904802 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:24.904818 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:24.905121 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:24.905532 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:24.905535 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:24.905537 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:24.905775 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:24.906241 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:24.906251 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:24.906571 (+ 320us) client_negotiation.cc:769] Sending connection context
0929 18:35:24.906775 (+ 204us) client_negotiation.cc:240] Negotiation successful
0929 18:35:24.907196 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":158,"threads_started":1}
I20260929 18:35:24.910522 12891 heartbeater.cc:347] Connected to a master server at 127.28.207.126:36041
I20260929 18:35:24.910936 12891 heartbeater.cc:464] Registering TS with master...
I20260929 18:35:24.911567 12891 heartbeater.cc:511] Master 127.28.207.126:36041 requested a full tablet report, sending...
I20260929 18:35:24.912943 12431 ts_manager.cc:194] Registered new tserver with Master: 74087a8bd66b403d8c9a22b68b090fd7 (127.28.207.67:35193)
I20260929 18:35:24.913781 12431 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.67@KRBTEST.COM'} at 127.28.207.67:54811
I20260929 18:35:24.922818 29501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:35:24.934090 12471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:24.925100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38694 (local address 127.28.207.126:36041)
0929 18:35:24.925327 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:24.925333 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:24.925514 (+ 181us) server_negotiation.cc:407] Connection header received
0929 18:35:24.925708 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:24.925711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:24.925785 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:24.925883 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:24.926703 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:24.927360 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:24.928244 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:24.928480 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:24.928595 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:24.930168 (+ 1573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:24.930173 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:24.930176 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:24.930209 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:24.931979 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:24.932483 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:24.932547 (+ 64us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:24.932549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:24.932646 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:24.932979 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:24.932983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:24.932985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:24.933276 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:24.933442 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:24.933699 (+ 257us) server_negotiation.cc:299] Negotiation successful
0929 18:35:24.933883 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260929 18:35:24.938771 12431 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38694:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:35:24.942759 12431 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:35:24.961452 12903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:24.953937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35820 (local address 127.28.207.67:35193)
0929 18:35:24.955010 (+ 1073us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:24.955026 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:24.955055 (+ 29us) server_negotiation.cc:407] Connection header received
0929 18:35:24.955159 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:24.955166 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:24.955477 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:24.955759 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:24.956157 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:24.957743 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:24.959552 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:24.960507 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:24.960666 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:24.960857 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:24.961058 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 18:35:24.961305 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":938,"thread_start_us":153,"threads_started":1}
I20260929 18:35:24.961440 12902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:24.953603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:35193 (local address 127.0.0.1:35820)
0929 18:35:24.954218 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:24.954314 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:24.954485 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:24.955739 (+ 1254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:24.955746 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:24.955772 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:24.956032 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:24.956041 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:24.958010 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:24.958014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:24.959414 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:24.959425 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:24.960727 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:24.960730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:24.960819 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:24.960860 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 18:35:24.961016 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 18:35:24.961263 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":120,"thread_start_us":139,"threads_started":1}
I20260929 18:35:24.962620 12904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:24.954475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:46723 (local address 127.0.0.1:36042)
0929 18:35:24.955313 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:24.955325 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:24.955412 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:24.956006 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:24.956008 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:24.956021 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:24.956106 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:24.956115 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:24.957752 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:24.957756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:24.959266 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:24.959276 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:24.962177 (+ 2901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:24.962181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:24.962278 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:24.962304 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 18:35:24.962383 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 18:35:24.962466 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"mutex_wait_us":10,"thread_start_us":180,"threads_started":1}
I20260929 18:35:24.962925 12906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:24.954909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36042 (local address 127.28.207.65:46723)
0929 18:35:24.955304 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:24.955313 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:24.955397 (+ 84us) server_negotiation.cc:407] Connection header received
0929 18:35:24.955517 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:24.955521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:24.955682 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:24.955855 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:24.956237 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:24.957596 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:24.959393 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:24.960508 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:24.960666 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:24.960857 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:24.962618 (+ 1761us) server_negotiation.cc:299] Negotiation successful
0929 18:35:24.962755 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":90,"threads_started":1}
I20260929 18:35:24.962951 12905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:24.954438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:46547 (local address 127.0.0.1:34954)
0929 18:35:24.955047 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:24.955061 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:24.955148 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:24.955867 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:24.955872 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:24.955894 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:24.956061 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:24.956069 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:24.957753 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:24.957756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:24.959414 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:24.959425 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:24.962374 (+ 2949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:24.962379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:24.962525 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:24.962561 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 18:35:24.962624 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 18:35:24.962721 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":69,"thread_start_us":99,"threads_started":1}
I20260929 18:35:24.963573 12907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:24.954921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34954 (local address 127.28.207.66:46547)
0929 18:35:24.955304 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:24.955318 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:24.955345 (+ 27us) server_negotiation.cc:407] Connection header received
0929 18:35:24.955433 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:24.955439 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:24.955692 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:24.956212 (+ 520us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:24.956382 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:24.957596 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:24.960982 (+ 3386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:24.962197 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:24.963035 (+ 838us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:24.963240 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:24.963300 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 18:35:24.963419 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":99,"threads_started":1}
I20260929 18:35:24.965935 12557 tablet_service.cc:1467] Processing CreateTablet for tablet 2a3db3d898a3430b85e0bcfff98afcb9 (DEFAULT_TABLE table=test-table [id=95f4b43dc6a44e53b9fb8c6756de7258]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:24.965935 12825 tablet_service.cc:1467] Processing CreateTablet for tablet 2a3db3d898a3430b85e0bcfff98afcb9 (DEFAULT_TABLE table=test-table [id=95f4b43dc6a44e53b9fb8c6756de7258]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:24.966343 12691 tablet_service.cc:1467] Processing CreateTablet for tablet 2a3db3d898a3430b85e0bcfff98afcb9 (DEFAULT_TABLE table=test-table [id=95f4b43dc6a44e53b9fb8c6756de7258]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:24.967487 12557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a3db3d898a3430b85e0bcfff98afcb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:24.967487 12825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a3db3d898a3430b85e0bcfff98afcb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:24.967602 12691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a3db3d898a3430b85e0bcfff98afcb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:24.975013 12908 tablet_bootstrap.cc:492] T 2a3db3d898a3430b85e0bcfff98afcb9 P 12e32b320f0349d18c5c4811cae01a0d: Bootstrap starting.
I20260929 18:35:24.978073 12909 tablet_bootstrap.cc:492] T 2a3db3d898a3430b85e0bcfff98afcb9 P 9dfe7e3bf3674eed89c366f59653a594: Bootstrap starting.
I20260929 18:35:24.978474 12908 tablet_bootstrap.cc:654] T 2a3db3d898a3430b85e0bcfff98afcb9 P 12e32b320f0349d18c5c4811cae01a0d: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:24.979566 12908 log.cc:824] T 2a3db3d898a3430b85e0bcfff98afcb9 P 12e32b320f0349d18c5c4811cae01a0d: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:24.979650 12910 tablet_bootstrap.cc:492] T 2a3db3d898a3430b85e0bcfff98afcb9 P 74087a8bd66b403d8c9a22b68b090fd7: Bootstrap starting.
I20260929 18:35:24.980851 12909 tablet_bootstrap.cc:654] T 2a3db3d898a3430b85e0bcfff98afcb9 P 9dfe7e3bf3674eed89c366f59653a594: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:24.981698 12909 log.cc:824] T 2a3db3d898a3430b85e0bcfff98afcb9 P 9dfe7e3bf3674eed89c366f59653a594: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:24.982208 12908 tablet_bootstrap.cc:492] T 2a3db3d898a3430b85e0bcfff98afcb9 P 12e32b320f0349d18c5c4811cae01a0d: No bootstrap required, opened a new log
I20260929 18:35:24.982478 12908 ts_tablet_manager.cc:1397] T 2a3db3d898a3430b85e0bcfff98afcb9 P 12e32b320f0349d18c5c4811cae01a0d: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260929 18:35:24.982661 12910 tablet_bootstrap.cc:654] T 2a3db3d898a3430b85e0bcfff98afcb9 P 74087a8bd66b403d8c9a22b68b090fd7: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:24.983574 12910 log.cc:824] T 2a3db3d898a3430b85e0bcfff98afcb9 P 74087a8bd66b403d8c9a22b68b090fd7: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:24.984210 12909 tablet_bootstrap.cc:492] T 2a3db3d898a3430b85e0bcfff98afcb9 P 9dfe7e3bf3674eed89c366f59653a594: No bootstrap required, opened a new log
I20260929 18:35:24.984403 12909 ts_tablet_manager.cc:1397] T 2a3db3d898a3430b85e0bcfff98afcb9 P 9dfe7e3bf3674eed89c366f59653a594: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 18:35:24.985746 12910 tablet_bootstrap.cc:492] T 2a3db3d898a3430b85e0bcfff98afcb9 P 74087a8bd66b403d8c9a22b68b090fd7: No bootstrap required, opened a new log
I20260929 18:35:24.985952 12910 ts_tablet_manager.cc:1397] T 2a3db3d898a3430b85e0bcfff98afcb9 P 74087a8bd66b403d8c9a22b68b090fd7: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 18:35:24.987144 12908 raft_consensus.cc:348] T 2a3db3d898a3430b85e0bcfff98afcb9 P 12e32b320f0349d18c5c4811cae01a0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74087a8bd66b403d8c9a22b68b090fd7" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 35193 } } peers { permanent_uuid: "12e32b320f0349d18c5c4811cae01a0d" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46547 } } peers { permanent_uuid: "9dfe7e3bf3674eed89c366f59653a594" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46723 } }
I20260929 18:35:24.987473 12908 raft_consensus.cc:374] T 2a3db3d898a3430b85e0bcfff98afcb9 P 12e32b320f0349d18c5c4811cae01a0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:24.987532 12908 raft_consensus.cc:729] T 2a3db3d898a3430b85e0bcfff98afcb9 P 12e32b320f0349d18c5c4811cae01a0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e32b320f0349d18c5c4811cae01a0d, State: Initialized, Role: FOLLOWER
I20260929 18:35:24.988134 12908 consensus_queue.cc:260] T 2a3db3d898a3430b85e0bcfff98afcb9 P 12e32b320f0349d18c5c4811cae01a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74087a8bd66b403d8c9a22b68b090fd7" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 35193 } } peers { permanent_uuid: "12e32b320f0349d18c5c4811cae01a0d" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46547 } } peers { permanent_uuid: "9dfe7e3bf3674eed89c366f59653a594" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46723 } }
I20260929 18:35:24.989636 12908 ts_tablet_manager.cc:1428] T 2a3db3d898a3430b85e0bcfff98afcb9 P 12e32b320f0349d18c5c4811cae01a0d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 18:35:24.990231 12757 heartbeater.cc:503] Master 127.28.207.126:36041 was elected leader, sending a full tablet report...
I20260929 18:35:24.990286 12909 raft_consensus.cc:348] T 2a3db3d898a3430b85e0bcfff98afcb9 P 9dfe7e3bf3674eed89c366f59653a594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74087a8bd66b403d8c9a22b68b090fd7" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 35193 } } peers { permanent_uuid: "12e32b320f0349d18c5c4811cae01a0d" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46547 } } peers { permanent_uuid: "9dfe7e3bf3674eed89c366f59653a594" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46723 } }
I20260929 18:35:24.990643 12909 raft_consensus.cc:374] T 2a3db3d898a3430b85e0bcfff98afcb9 P 9dfe7e3bf3674eed89c366f59653a594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:24.990695 12909 raft_consensus.cc:729] T 2a3db3d898a3430b85e0bcfff98afcb9 P 9dfe7e3bf3674eed89c366f59653a594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dfe7e3bf3674eed89c366f59653a594, State: Initialized, Role: FOLLOWER
I20260929 18:35:24.991191 12909 consensus_queue.cc:260] T 2a3db3d898a3430b85e0bcfff98afcb9 P 9dfe7e3bf3674eed89c366f59653a594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74087a8bd66b403d8c9a22b68b090fd7" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 35193 } } peers { permanent_uuid: "12e32b320f0349d18c5c4811cae01a0d" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46547 } } peers { permanent_uuid: "9dfe7e3bf3674eed89c366f59653a594" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46723 } }
I20260929 18:35:24.991470 12910 raft_consensus.cc:348] T 2a3db3d898a3430b85e0bcfff98afcb9 P 74087a8bd66b403d8c9a22b68b090fd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74087a8bd66b403d8c9a22b68b090fd7" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 35193 } } peers { permanent_uuid: "12e32b320f0349d18c5c4811cae01a0d" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46547 } } peers { permanent_uuid: "9dfe7e3bf3674eed89c366f59653a594" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46723 } }
I20260929 18:35:24.991845 12910 raft_consensus.cc:374] T 2a3db3d898a3430b85e0bcfff98afcb9 P 74087a8bd66b403d8c9a22b68b090fd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:24.991964 12910 raft_consensus.cc:729] T 2a3db3d898a3430b85e0bcfff98afcb9 P 74087a8bd66b403d8c9a22b68b090fd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74087a8bd66b403d8c9a22b68b090fd7, State: Initialized, Role: FOLLOWER
I20260929 18:35:24.992250 12909 ts_tablet_manager.cc:1428] T 2a3db3d898a3430b85e0bcfff98afcb9 P 9dfe7e3bf3674eed89c366f59653a594: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260929 18:35:24.992529 12910 consensus_queue.cc:260] T 2a3db3d898a3430b85e0bcfff98afcb9 P 74087a8bd66b403d8c9a22b68b090fd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74087a8bd66b403d8c9a22b68b090fd7" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 35193 } } peers { permanent_uuid: "12e32b320f0349d18c5c4811cae01a0d" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46547 } } peers { permanent_uuid: "9dfe7e3bf3674eed89c366f59653a594" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46723 } }
I20260929 18:35:24.992641 12623 heartbeater.cc:503] Master 127.28.207.126:36041 was elected leader, sending a full tablet report...
I20260929 18:35:24.993539 12891 heartbeater.cc:503] Master 127.28.207.126:36041 was elected leader, sending a full tablet report...
I20260929 18:35:24.994246 12910 ts_tablet_manager.cc:1428] T 2a3db3d898a3430b85e0bcfff98afcb9 P 74087a8bd66b403d8c9a22b68b090fd7: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260929 18:35:25.031304 12916 raft_consensus.cc:482] T 2a3db3d898a3430b85e0bcfff98afcb9 P 74087a8bd66b403d8c9a22b68b090fd7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:35:25.031570 12916 raft_consensus.cc:504] T 2a3db3d898a3430b85e0bcfff98afcb9 P 74087a8bd66b403d8c9a22b68b090fd7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74087a8bd66b403d8c9a22b68b090fd7" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 35193 } } peers { permanent_uuid: "12e32b320f0349d18c5c4811cae01a0d" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46547 } } peers { permanent_uuid: "9dfe7e3bf3674eed89c366f59653a594" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46723 } }
I20260929 18:35:25.032897 12916 leader_election.cc:288] T 2a3db3d898a3430b85e0bcfff98afcb9 P 74087a8bd66b403d8c9a22b68b090fd7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12e32b320f0349d18c5c4811cae01a0d (127.28.207.66:46547), 9dfe7e3bf3674eed89c366f59653a594 (127.28.207.65:46723)
I20260929 18:35:25.037398 12893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:25.033294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:46723 (local address 127.28.207.67:42849)
0929 18:35:25.033468 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:25.033498 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:25.033640 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:25.034023 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:25.034027 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:25.034045 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:25.034192 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:25.034198 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:25.035032 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:25.035039 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:25.035896 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:25.035904 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:25.036750 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:25.036754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:25.036885 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:25.036935 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 18:35:25.037017 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 18:35:25.037110 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 18:35:25.037576 12906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:25.033413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:42849 (local address 127.28.207.65:46723)
0929 18:35:25.033540 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:25.033546 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:25.033567 (+ 21us) server_negotiation.cc:407] Connection header received
0929 18:35:25.033851 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:25.033855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:25.033910 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:25.034000 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:25.034306 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:25.034928 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:25.036029 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:25.036626 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:25.036766 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:25.036833 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:25.037227 (+ 394us) server_negotiation.cc:299] Negotiation successful
0929 18:35:25.037425 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 18:35:25.037611 12907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:25.033620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:55667 (local address 127.28.207.66:46547)
0929 18:35:25.033771 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:25.033776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:25.033946 (+ 170us) server_negotiation.cc:407] Connection header received
0929 18:35:25.034127 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:25.034130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:25.034186 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:25.034278 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:25.034579 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:25.035176 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:25.036447 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:25.037018 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:25.037151 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:25.037209 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:25.037394 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 18:35:25.037464 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 18:35:25.037773 12917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:25.033342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:46547 (local address 127.28.207.67:55667)
0929 18:35:25.033827 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:25.033850 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:25.033999 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:25.034288 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:25.034291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:25.034307 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:25.034459 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:25.034465 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:25.035285 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:25.035291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:25.036326 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:25.036336 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:25.037152 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:25.037155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:25.037257 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:25.037283 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 18:35:25.037379 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 18:35:25.037461 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":211,"threads_started":1}
I20260929 18:35:25.038309 12577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a3db3d898a3430b85e0bcfff98afcb9" candidate_uuid: "74087a8bd66b403d8c9a22b68b090fd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dfe7e3bf3674eed89c366f59653a594" is_pre_election: true
I20260929 18:35:25.038486 12711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a3db3d898a3430b85e0bcfff98afcb9" candidate_uuid: "74087a8bd66b403d8c9a22b68b090fd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e32b320f0349d18c5c4811cae01a0d" is_pre_election: true
I20260929 18:35:25.038693 12577 raft_consensus.cc:2454] T 2a3db3d898a3430b85e0bcfff98afcb9 P 9dfe7e3bf3674eed89c366f59653a594 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74087a8bd66b403d8c9a22b68b090fd7 in term 0.
I20260929 18:35:25.038743 12711 raft_consensus.cc:2454] T 2a3db3d898a3430b85e0bcfff98afcb9 P 12e32b320f0349d18c5c4811cae01a0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74087a8bd66b403d8c9a22b68b090fd7 in term 0.
I20260929 18:35:25.039347 12778 leader_election.cc:302] T 2a3db3d898a3430b85e0bcfff98afcb9 P 74087a8bd66b403d8c9a22b68b090fd7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 12e32b320f0349d18c5c4811cae01a0d, 74087a8bd66b403d8c9a22b68b090fd7, 9dfe7e3bf3674eed89c366f59653a594; no voters:
I20260929 18:35:25.039698 12916 raft_consensus.cc:2781] T 2a3db3d898a3430b85e0bcfff98afcb9 P 74087a8bd66b403d8c9a22b68b090fd7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:35:25.039831 12916 raft_consensus.cc:482] T 2a3db3d898a3430b85e0bcfff98afcb9 P 74087a8bd66b403d8c9a22b68b090fd7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:35:25.039899 12916 raft_consensus.cc:3037] T 2a3db3d898a3430b85e0bcfff98afcb9 P 74087a8bd66b403d8c9a22b68b090fd7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:25.041451 12916 raft_consensus.cc:504] T 2a3db3d898a3430b85e0bcfff98afcb9 P 74087a8bd66b403d8c9a22b68b090fd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74087a8bd66b403d8c9a22b68b090fd7" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 35193 } } peers { permanent_uuid: "12e32b320f0349d18c5c4811cae01a0d" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46547 } } peers { permanent_uuid: "9dfe7e3bf3674eed89c366f59653a594" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46723 } }
I20260929 18:35:25.041936 12916 leader_election.cc:288] T 2a3db3d898a3430b85e0bcfff98afcb9 P 74087a8bd66b403d8c9a22b68b090fd7 [CANDIDATE]: Term 1 election: Requested vote from peers 12e32b320f0349d18c5c4811cae01a0d (127.28.207.66:46547), 9dfe7e3bf3674eed89c366f59653a594 (127.28.207.65:46723)
I20260929 18:35:25.042496 12577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a3db3d898a3430b85e0bcfff98afcb9" candidate_uuid: "74087a8bd66b403d8c9a22b68b090fd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dfe7e3bf3674eed89c366f59653a594"
I20260929 18:35:25.042496 12711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a3db3d898a3430b85e0bcfff98afcb9" candidate_uuid: "74087a8bd66b403d8c9a22b68b090fd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e32b320f0349d18c5c4811cae01a0d"
I20260929 18:35:25.042665 12577 raft_consensus.cc:3037] T 2a3db3d898a3430b85e0bcfff98afcb9 P 9dfe7e3bf3674eed89c366f59653a594 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:25.042663 12711 raft_consensus.cc:3037] T 2a3db3d898a3430b85e0bcfff98afcb9 P 12e32b320f0349d18c5c4811cae01a0d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:25.044147 12577 raft_consensus.cc:2454] T 2a3db3d898a3430b85e0bcfff98afcb9 P 9dfe7e3bf3674eed89c366f59653a594 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74087a8bd66b403d8c9a22b68b090fd7 in term 1.
I20260929 18:35:25.044147 12711 raft_consensus.cc:2454] T 2a3db3d898a3430b85e0bcfff98afcb9 P 12e32b320f0349d18c5c4811cae01a0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74087a8bd66b403d8c9a22b68b090fd7 in term 1.
I20260929 18:35:25.044557 12781 leader_election.cc:302] T 2a3db3d898a3430b85e0bcfff98afcb9 P 74087a8bd66b403d8c9a22b68b090fd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74087a8bd66b403d8c9a22b68b090fd7, 9dfe7e3bf3674eed89c366f59653a594; no voters:
I20260929 18:35:25.044833 12916 raft_consensus.cc:2781] T 2a3db3d898a3430b85e0bcfff98afcb9 P 74087a8bd66b403d8c9a22b68b090fd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:25.045171 12916 raft_consensus.cc:686] T 2a3db3d898a3430b85e0bcfff98afcb9 P 74087a8bd66b403d8c9a22b68b090fd7 [term 1 LEADER]: Becoming Leader. State: Replica: 74087a8bd66b403d8c9a22b68b090fd7, State: Running, Role: LEADER
I20260929 18:35:25.045668 12916 consensus_queue.cc:237] T 2a3db3d898a3430b85e0bcfff98afcb9 P 74087a8bd66b403d8c9a22b68b090fd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74087a8bd66b403d8c9a22b68b090fd7" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 35193 } } peers { permanent_uuid: "12e32b320f0349d18c5c4811cae01a0d" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46547 } } peers { permanent_uuid: "9dfe7e3bf3674eed89c366f59653a594" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46723 } }
I20260929 18:35:25.049490 12430 catalog_manager.cc:5495] T 2a3db3d898a3430b85e0bcfff98afcb9 P 74087a8bd66b403d8c9a22b68b090fd7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74087a8bd66b403d8c9a22b68b090fd7 (127.28.207.67). New cstate: current_term: 1 leader_uuid: "74087a8bd66b403d8c9a22b68b090fd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74087a8bd66b403d8c9a22b68b090fd7" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 35193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12e32b320f0349d18c5c4811cae01a0d" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dfe7e3bf3674eed89c366f59653a594" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46723 } health_report { overall_health: UNKNOWN } } }
I20260929 18:35:25.066308 12903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:25.062286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35832 (local address 127.28.207.67:35193)
0929 18:35:25.062427 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:25.062430 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:25.062446 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:35:25.062617 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:25.062620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:25.062676 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:25.062781 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:35:25.063183 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:25.063728 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:25.064793 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:25.065023 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:25.065167 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:25.065463 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:35:25.065881 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:35:25.066009 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:25.066071 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 18:35:25.066138 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:35:25.073573 12711 raft_consensus.cc:1261] T 2a3db3d898a3430b85e0bcfff98afcb9 P 12e32b320f0349d18c5c4811cae01a0d [term 1 FOLLOWER]: Refusing update from remote peer 74087a8bd66b403d8c9a22b68b090fd7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:35:25.073573 12577 raft_consensus.cc:1261] T 2a3db3d898a3430b85e0bcfff98afcb9 P 9dfe7e3bf3674eed89c366f59653a594 [term 1 FOLLOWER]: Refusing update from remote peer 74087a8bd66b403d8c9a22b68b090fd7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:35:25.074517 12918 consensus_queue.cc:1035] T 2a3db3d898a3430b85e0bcfff98afcb9 P 74087a8bd66b403d8c9a22b68b090fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dfe7e3bf3674eed89c366f59653a594" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:35:25.074793 12916 consensus_queue.cc:1035] T 2a3db3d898a3430b85e0bcfff98afcb9 P 74087a8bd66b403d8c9a22b68b090fd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e32b320f0349d18c5c4811cae01a0d" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:35:25.083675 12921 mvcc.cc:205] Tried to move back new op lower bound from 7334735565091287040 to 7334735564990410752. Current Snapshot: MvccSnapshot[applied={T|T < 7334735565091287040}]
I20260929 18:35:25.085352 12922 mvcc.cc:205] Tried to move back new op lower bound from 7334735565091287040 to 7334735564990410752. Current Snapshot: MvccSnapshot[applied={T|T < 7334735565091287040}]
I20260929 18:35:25.085229 12923 mvcc.cc:205] Tried to move back new op lower bound from 7334735565091287040 to 7334735564990410752. Current Snapshot: MvccSnapshot[applied={T|T < 7334735565091287040}]
I20260929 18:35:25.095080 12430 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:35:25.095291 12430 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:35:25.098459 12430 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46: Sending DeleteTablet for 3 replicas of tablet 2a3db3d898a3430b85e0bcfff98afcb9
I20260929 18:35:25.099681 12825 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a3db3d898a3430b85e0bcfff98afcb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:25 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:35820
I20260929 18:35:25.099692 12691 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a3db3d898a3430b85e0bcfff98afcb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:25 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:34954
I20260929 18:35:25.100565 12930 tablet_replica.cc:330] T 2a3db3d898a3430b85e0bcfff98afcb9 P 12e32b320f0349d18c5c4811cae01a0d: stopping tablet replica
I20260929 18:35:25.100806 12557 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a3db3d898a3430b85e0bcfff98afcb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:25 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:36042
I20260929 18:35:25.100980 12930 raft_consensus.cc:2229] T 2a3db3d898a3430b85e0bcfff98afcb9 P 12e32b320f0349d18c5c4811cae01a0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:35:25.101359 12930 raft_consensus.cc:2258] T 2a3db3d898a3430b85e0bcfff98afcb9 P 12e32b320f0349d18c5c4811cae01a0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:35:25.101243 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 12494
I20260929 18:35:25.101475 12932 tablet_replica.cc:330] T 2a3db3d898a3430b85e0bcfff98afcb9 P 9dfe7e3bf3674eed89c366f59653a594: stopping tablet replica
I20260929 18:35:25.101783 12931 tablet_replica.cc:330] T 2a3db3d898a3430b85e0bcfff98afcb9 P 74087a8bd66b403d8c9a22b68b090fd7: stopping tablet replica
I20260929 18:35:25.102074 12931 raft_consensus.cc:2229] T 2a3db3d898a3430b85e0bcfff98afcb9 P 74087a8bd66b403d8c9a22b68b090fd7 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:35:25.102499 12931 raft_consensus.cc:2258] T 2a3db3d898a3430b85e0bcfff98afcb9 P 74087a8bd66b403d8c9a22b68b090fd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:35:25.103945 12931 ts_tablet_manager.cc:1905] T 2a3db3d898a3430b85e0bcfff98afcb9 P 74087a8bd66b403d8c9a22b68b090fd7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:35:25.103945 12930 ts_tablet_manager.cc:1905] T 2a3db3d898a3430b85e0bcfff98afcb9 P 12e32b320f0349d18c5c4811cae01a0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:35:25.107033 12930 ts_tablet_manager.cc:1918] T 2a3db3d898a3430b85e0bcfff98afcb9 P 12e32b320f0349d18c5c4811cae01a0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:35:25.107239 12930 log.cc:1196] T 2a3db3d898a3430b85e0bcfff98afcb9 P 12e32b320f0349d18c5c4811cae01a0d: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/wal/wals/2a3db3d898a3430b85e0bcfff98afcb9
I20260929 18:35:25.107301 12931 ts_tablet_manager.cc:1918] T 2a3db3d898a3430b85e0bcfff98afcb9 P 74087a8bd66b403d8c9a22b68b090fd7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:35:25.107398 12931 log.cc:1196] T 2a3db3d898a3430b85e0bcfff98afcb9 P 74087a8bd66b403d8c9a22b68b090fd7: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/wal/wals/2a3db3d898a3430b85e0bcfff98afcb9
I20260929 18:35:25.107800 12930 ts_tablet_manager.cc:1939] T 2a3db3d898a3430b85e0bcfff98afcb9 P 12e32b320f0349d18c5c4811cae01a0d: Deleting consensus metadata
I20260929 18:35:25.107800 12931 ts_tablet_manager.cc:1939] T 2a3db3d898a3430b85e0bcfff98afcb9 P 74087a8bd66b403d8c9a22b68b090fd7: Deleting consensus metadata
I20260929 18:35:25.109306 12416 catalog_manager.cc:4841] TS 12e32b320f0349d18c5c4811cae01a0d (127.28.207.66:46547): tablet 2a3db3d898a3430b85e0bcfff98afcb9 (table test-table [id=95f4b43dc6a44e53b9fb8c6756de7258]) successfully deleted
I20260929 18:35:25.109598 12416 catalog_manager.cc:4841] TS 74087a8bd66b403d8c9a22b68b090fd7 (127.28.207.67:35193): tablet 2a3db3d898a3430b85e0bcfff98afcb9 (table test-table [id=95f4b43dc6a44e53b9fb8c6756de7258]) successfully deleted
W20260929 18:35:25.112819 12419 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.207.65:46723 (error 108)
I20260929 18:35:25.113556 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 12628
I20260929 18:35:25.115947 12905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:25.115585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:46723 (local address 127.0.0.1:36048)
0929 18:35:25.115759 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:25.115847 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.65:46723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260929 18:35:25.116180 12419 catalog_manager.cc:4584] TS 9dfe7e3bf3674eed89c366f59653a594 (127.28.207.65:46723): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2a3db3d898a3430b85e0bcfff98afcb9: Network error: Client connection negotiation failed: client connection to 127.28.207.65:46723: connect: Connection refused (error 111)
I20260929 18:35:25.121515 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 12762
I20260929 18:35:25.127790 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 12400
I20260929 18:35:45.135326 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:36041
--webserver_interface=127.28.207.126
--webserver_port=36549
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:42001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:36041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:35:45.260524 12934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:45.260824 12934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:45.260869 12934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:45.264953 12934 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 18:35:45.265021 12934 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 18:35:45.265038 12934 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 18:35:45.265053 12934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:35:45.265071 12934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:45.265090 12934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:45.265106 12934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:35:45.265121 12934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:35:45.270699 12934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:42001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:36041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:36041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=36549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:35:45.271821 12934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:45.272833 12934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:45.287292 12943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:45.287444 12934 server_base.cc:1034] running on GCE node
W20260929 18:35:45.287300 12940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:45.287300 12941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:45.288370 12934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:45.289569 12934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:45.290771 12934 hybrid_clock.cc:648] HybridClock initialized: now 1790706945290751 us; error 41 us; skew 500 ppm
Sep 29 18:35:45 dist-test-slave-qcpv krb5kdc[12387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706945, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:45.294020 12934 init.cc:377] Logged in from keytab as kudu/127.28.207.126@KRBTEST.COM (short username kudu)
I20260929 18:35:45.295110 12934 webserver.cc:460] Webserver started at http://127.28.207.126:36549/ using document root <none> and password file <none>
I20260929 18:35:45.295740 12934 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:45.295818 12934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:45.300053 12934 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:35:45.302481 12950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:45.303973 12934 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:35:45.304132 12934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "30e2464e79e64d83a9d4b044ec3e3a46"
format_stamp: "Formatted at 2026-09-29 18:35:23 on dist-test-slave-qcpv"
server_key: "42857d24f67d8174476fabdc8ede8779"
server_key_iv: "29b71d3de15d499b25bb7a1c75d33bd4"
server_key_version: "encryptionkey@0"
I20260929 18:35:45.304735 12934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:45.339908 12934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:45.343436 12934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:45.343642 12934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:45.351675 12934 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:36041
I20260929 18:35:45.352138 13002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:36041 every 8 connection(s)
I20260929 18:35:45.352871 12934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:35:45.353762 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 12934
I20260929 18:35:45.354414 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.65:46723
--local_ip_for_outbound_sockets=127.28.207.65
--tserver_master_addrs=127.28.207.126:36041
--webserver_port=34605
--webserver_interface=127.28.207.65
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:42001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:35:45.359876 13003 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:35:45.361994 13003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46: Bootstrap starting.
I20260929 18:35:45.383936 13003 log.cc:824] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:45.394205 13003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:35:45.394734 13003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46: Bootstrap complete.
I20260929 18:35:45.399389 13003 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e2464e79e64d83a9d4b044ec3e3a46" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 36041 } }
I20260929 18:35:45.399912 13003 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30e2464e79e64d83a9d4b044ec3e3a46, State: Initialized, Role: FOLLOWER
I20260929 18:35:45.400538 13003 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e2464e79e64d83a9d4b044ec3e3a46" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 36041 } }
I20260929 18:35:45.400671 13003 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:35:45.400769 13003 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:35:45.400887 13003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [term 1 FOLLOWER]: Advancing to term 2
I20260929 18:35:45.403298 13003 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e2464e79e64d83a9d4b044ec3e3a46" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 36041 } }
I20260929 18:35:45.403683 13003 leader_election.cc:302] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30e2464e79e64d83a9d4b044ec3e3a46; no voters:
I20260929 18:35:45.404064 13003 leader_election.cc:288] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 18:35:45.404163 13007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 18:35:45.404399 13007 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [term 2 LEADER]: Becoming Leader. State: Replica: 30e2464e79e64d83a9d4b044ec3e3a46, State: Running, Role: LEADER
I20260929 18:35:45.404736 13007 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e2464e79e64d83a9d4b044ec3e3a46" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 36041 } }
I20260929 18:35:45.405520 13003 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:35:45.405838 13007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30e2464e79e64d83a9d4b044ec3e3a46. Latest consensus state: current_term: 2 leader_uuid: "30e2464e79e64d83a9d4b044ec3e3a46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e2464e79e64d83a9d4b044ec3e3a46" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 36041 } } }
I20260929 18:35:45.405961 13007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:45.406342 13011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:35:45.406436 13008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "30e2464e79e64d83a9d4b044ec3e3a46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e2464e79e64d83a9d4b044ec3e3a46" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 36041 } } }
I20260929 18:35:45.406709 13008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [sys.catalog]: This master's current role is: LEADER
I20260929 18:35:45.412366 13011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:35:45.413076 13011 catalog_manager.cc:1259] Loaded cluster ID: b8357524696649b287da1fcb2b016672
I20260929 18:35:45.413156 13011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:35:45.415246 13011 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:35:45.415876 13011 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46: Loaded TSK: 0
I20260929 18:35:45.432793 13011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46: Generated new TSK 1
I20260929 18:35:45.433101 13011 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 18:35:45.494663 13005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:45.495055 13005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:45.495143 13005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:45.499652 13005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:45.499744 13005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:45.499840 13005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.65
I20260929 18:35:45.505695 13005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:42001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.65:46723
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.207.65
--webserver_port=34605
--tserver_master_addrs=127.28.207.126:36041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.65
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:35:45.507185 13005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:45.508605 13005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:45.519903 13030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:45.520133 13005 server_base.cc:1034] running on GCE node
W20260929 18:35:45.519903 13031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:45.519959 13033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:45.520913 13005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:45.521703 13005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:45.522994 13005 hybrid_clock.cc:648] HybridClock initialized: now 1790706945522962 us; error 56 us; skew 500 ppm
Sep 29 18:35:45 dist-test-slave-qcpv krb5kdc[12387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706945, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:45.526789 13005 init.cc:377] Logged in from keytab as kudu/127.28.207.65@KRBTEST.COM (short username kudu)
I20260929 18:35:45.528187 13005 webserver.cc:460] Webserver started at http://127.28.207.65:34605/ using document root <none> and password file <none>
I20260929 18:35:45.528946 13005 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:45.529072 13005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:45.534387 13005 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:35:45.537415 13040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:45.539173 13005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 18:35:45.539327 13005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/wal
uuid: "9dfe7e3bf3674eed89c366f59653a594"
format_stamp: "Formatted at 2026-09-29 18:35:24 on dist-test-slave-qcpv"
server_key: "344cfc47f7a3d5e65f98bd25d917a8c4"
server_key_iv: "ba61fbd4b04313419210b3d5a5f13ac9"
server_key_version: "encryptionkey@0"
I20260929 18:35:45.540035 13005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:45.567492 13005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:45.571271 13005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:45.571480 13005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:45.572204 13005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:35:45.573562 13047 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 18:35:45.578758 13005 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 18:35:45.578848 13005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260929 18:35:45.578940 13005 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 18:35:45.581416 13005 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 18:35:45.581473 13005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260929 18:35:45.581947 13047 tablet_bootstrap.cc:492] T 2a3db3d898a3430b85e0bcfff98afcb9 P 9dfe7e3bf3674eed89c366f59653a594: Bootstrap starting.
I20260929 18:35:45.592367 13005 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.65:46723
I20260929 18:35:45.592926 13154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.65:46723 every 8 connection(s)
I20260929 18:35:45.593500 13005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
I20260929 18:35:45.603689 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 13005
I20260929 18:35:45.604343 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.66:46547
--local_ip_for_outbound_sockets=127.28.207.66
--tserver_master_addrs=127.28.207.126:36041
--webserver_port=44385
--webserver_interface=127.28.207.66
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:42001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:35:45 dist-test-slave-qcpv krb5kdc[12387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706945, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:35:45.608637 13047 log.cc:824] T 2a3db3d898a3430b85e0bcfff98afcb9 P 9dfe7e3bf3674eed89c366f59653a594: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:45.616636 13157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:45.596351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:41505 (local address 127.28.207.126:36041)
0929 18:35:45.596815 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:45.596839 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:45.597100 (+ 261us) server_negotiation.cc:407] Connection header received
0929 18:35:45.598384 (+ 1284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:45.598404 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:45.598941 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:45.599481 (+ 540us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:45.600663 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:45.602693 (+ 2030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:45.604305 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:45.605061 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:45.605286 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:45.609006 (+ 3720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:45.609016 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:45.609034 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:45.609093 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:45.611833 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:45.612664 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:45.612674 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:45.612681 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:45.612861 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:45.613383 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:45.613389 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:45.613392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:45.613971 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:45.614121 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:45.615069 (+ 948us) server_negotiation.cc:299] Negotiation successful
0929 18:35:45.615483 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":122,"threads_started":1}
I20260929 18:35:45.617669 13047 tablet_bootstrap.cc:492] T 2a3db3d898a3430b85e0bcfff98afcb9 P 9dfe7e3bf3674eed89c366f59653a594: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:35:45.618166 13047 tablet_bootstrap.cc:492] T 2a3db3d898a3430b85e0bcfff98afcb9 P 9dfe7e3bf3674eed89c366f59653a594: Bootstrap complete.
I20260929 18:35:45.618567 13158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:45.596223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.65:41505)
0929 18:35:45.596902 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:45.596955 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:45.597822 (+ 867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:45.599499 (+ 1677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:45.599506 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:45.599961 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:45.600461 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:45.600475 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:45.603010 (+ 2535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:45.603014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:45.604132 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:45.604144 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:45.605239 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:45.605246 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:45.605427 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:45.606212 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:45.606229 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:45.608596 (+ 2367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:45.612017 (+ 3421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:45.612028 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:45.612047 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:45.612489 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:45.613017 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:45.613022 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:45.613026 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:45.613242 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:45.614231 (+ 989us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:45.614242 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:45.614613 (+ 371us) client_negotiation.cc:769] Sending connection context
0929 18:35:45.617123 (+ 2510us) client_negotiation.cc:240] Negotiation successful
0929 18:35:45.617529 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":160,"threads_started":1}
I20260929 18:35:45.619619 13047 ts_tablet_manager.cc:1397] T 2a3db3d898a3430b85e0bcfff98afcb9 P 9dfe7e3bf3674eed89c366f59653a594: Time spent bootstrapping tablet: real 0.038s user 0.021s sys 0.015s
I20260929 18:35:45.621783 13155 heartbeater.cc:347] Connected to a master server at 127.28.207.126:36041
I20260929 18:35:45.622079 13155 heartbeater.cc:464] Registering TS with master...
I20260929 18:35:45.622778 13155 heartbeater.cc:511] Master 127.28.207.126:36041 requested a full tablet report, sending...
I20260929 18:35:45.622911 13047 raft_consensus.cc:348] T 2a3db3d898a3430b85e0bcfff98afcb9 P 9dfe7e3bf3674eed89c366f59653a594 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74087a8bd66b403d8c9a22b68b090fd7" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 35193 } } peers { permanent_uuid: "12e32b320f0349d18c5c4811cae01a0d" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46547 } } peers { permanent_uuid: "9dfe7e3bf3674eed89c366f59653a594" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46723 } }
I20260929 18:35:45.623546 13047 raft_consensus.cc:729] T 2a3db3d898a3430b85e0bcfff98afcb9 P 9dfe7e3bf3674eed89c366f59653a594 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dfe7e3bf3674eed89c366f59653a594, State: Initialized, Role: FOLLOWER
I20260929 18:35:45.624137 13047 consensus_queue.cc:260] T 2a3db3d898a3430b85e0bcfff98afcb9 P 9dfe7e3bf3674eed89c366f59653a594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74087a8bd66b403d8c9a22b68b090fd7" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 35193 } } peers { permanent_uuid: "12e32b320f0349d18c5c4811cae01a0d" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46547 } } peers { permanent_uuid: "9dfe7e3bf3674eed89c366f59653a594" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46723 } }
I20260929 18:35:45.625454 13047 ts_tablet_manager.cc:1428] T 2a3db3d898a3430b85e0bcfff98afcb9 P 9dfe7e3bf3674eed89c366f59653a594: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260929 18:35:45.627940 12967 ts_manager.cc:194] Registered new tserver with Master: 9dfe7e3bf3674eed89c366f59653a594 (127.28.207.65:46723)
I20260929 18:35:45.631841 12967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.65@KRBTEST.COM'} at 127.28.207.65:41505
I20260929 18:35:45.634713 13155 heartbeater.cc:503] Master 127.28.207.126:36041 was elected leader, sending a full tablet report...
Sep 29 18:35:45 dist-test-slave-qcpv krb5kdc[12387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706945, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for kudu/127.28.207.65@KRBTEST.COM
I20260929 18:35:45.643769 13163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:45.631465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60392 (local address 127.28.207.65:46723)
0929 18:35:45.632049 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:45.632065 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:45.632301 (+ 236us) server_negotiation.cc:407] Connection header received
0929 18:35:45.632526 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:45.632532 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:45.632764 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:45.632950 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:45.633756 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:45.634980 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:45.636114 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:45.636496 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:45.636632 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:45.639682 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:45.639696 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:45.639708 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:45.639732 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:45.641608 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:45.642188 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:45.642199 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:45.642201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:45.642280 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:45.642577 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:45.642580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:45.642581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:45.642897 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:45.643057 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:45.643405 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 18:35:45.643592 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":162,"threads_started":1}
I20260929 18:35:45.643769 13162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:45.631199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:46723 (local address 127.0.0.1:60392)
0929 18:35:45.632128 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:45.632177 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:45.632345 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:45.632966 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:45.632972 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:45.633461 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:45.633608 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:45.633617 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:45.635133 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:45.635137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:45.635971 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:45.635981 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:45.636634 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:45.636637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:45.636751 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:45.637482 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:45.637502 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:45.639525 (+ 2023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:45.641779 (+ 2254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:45.641789 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:45.641796 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:45.642061 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:45.642391 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:45.642395 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:45.642397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:45.642482 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:45.643018 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:45.643025 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:45.643166 (+ 141us) client_negotiation.cc:769] Sending connection context
0929 18:35:45.643316 (+ 150us) client_negotiation.cc:240] Negotiation successful
0929 18:35:45.643562 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"thread_start_us":141,"threads_started":1}
I20260929 18:35:45.645202 13089 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a3db3d898a3430b85e0bcfff98afcb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:25 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:60392
I20260929 18:35:45.646407 13164 tablet_replica.cc:330] stopping tablet replica
I20260929 18:35:45.646836 13164 raft_consensus.cc:2229] T 2a3db3d898a3430b85e0bcfff98afcb9 P 9dfe7e3bf3674eed89c366f59653a594 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:35:45.647193 13164 raft_consensus.cc:2258] T 2a3db3d898a3430b85e0bcfff98afcb9 P 9dfe7e3bf3674eed89c366f59653a594 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:35:45.648461 13164 ts_tablet_manager.cc:1905] T 2a3db3d898a3430b85e0bcfff98afcb9 P 9dfe7e3bf3674eed89c366f59653a594: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:35:45.653391 13164 ts_tablet_manager.cc:1918] T 2a3db3d898a3430b85e0bcfff98afcb9 P 9dfe7e3bf3674eed89c366f59653a594: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:35:45.653517 13164 log.cc:1196] T 2a3db3d898a3430b85e0bcfff98afcb9 P 9dfe7e3bf3674eed89c366f59653a594: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/wal/wals/2a3db3d898a3430b85e0bcfff98afcb9
I20260929 18:35:45.653820 13164 ts_tablet_manager.cc:1939] T 2a3db3d898a3430b85e0bcfff98afcb9 P 9dfe7e3bf3674eed89c366f59653a594: Deleting consensus metadata
I20260929 18:35:45.656116 12954 catalog_manager.cc:4841] TS 9dfe7e3bf3674eed89c366f59653a594 (127.28.207.65:46723): tablet 2a3db3d898a3430b85e0bcfff98afcb9 (table test-table [id=95f4b43dc6a44e53b9fb8c6756de7258]) successfully deleted
W20260929 18:35:45.732671 13159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:45.733044 13159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:45.733129 13159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:45.738276 13159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:45.738354 13159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:45.738416 13159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.66
I20260929 18:35:45.743969 13159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:42001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.66:46547
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.207.66
--webserver_port=44385
--tserver_master_addrs=127.28.207.126:36041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.66
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:35:45.745139 13159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:45.746297 13159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:45.756731 13170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:45.757079 13172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:45.756742 13169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:45.757216 13159 server_base.cc:1034] running on GCE node
I20260929 18:35:45.757795 13159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:45.758617 13159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:45.759920 13159 hybrid_clock.cc:648] HybridClock initialized: now 1790706945759913 us; error 53 us; skew 500 ppm
Sep 29 18:35:45 dist-test-slave-qcpv krb5kdc[12387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706945, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:45.763278 13159 init.cc:377] Logged in from keytab as kudu/127.28.207.66@KRBTEST.COM (short username kudu)
I20260929 18:35:45.764518 13159 webserver.cc:460] Webserver started at http://127.28.207.66:44385/ using document root <none> and password file <none>
I20260929 18:35:45.765164 13159 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:45.765244 13159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:45.770116 13159 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:35:45.772833 13179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:45.774669 13159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:35:45.774809 13159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/wal
uuid: "12e32b320f0349d18c5c4811cae01a0d"
format_stamp: "Formatted at 2026-09-29 18:35:24 on dist-test-slave-qcpv"
server_key: "501e0055f9101ec05c9cb1cde80eca59"
server_key_iv: "2c9fd2e5322082e5eaef49d2d004141f"
server_key_version: "encryptionkey@0"
I20260929 18:35:45.775641 13159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:45.806228 13159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:45.810719 13159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:45.811009 13159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:45.811817 13159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:35:45.812768 13159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:35:45.812819 13159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:45.812865 13159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:35:45.812879 13159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:45.824828 13159 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.66:46547
I20260929 18:35:45.824992 13292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.66:46547 every 8 connection(s)
I20260929 18:35:45.826179 13159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
I20260929 18:35:45.835260 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 13159
I20260929 18:35:45.835819 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.67:35193
--local_ip_for_outbound_sockets=127.28.207.67
--tserver_master_addrs=127.28.207.126:36041
--webserver_port=36799
--webserver_interface=127.28.207.67
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:42001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:35:45 dist-test-slave-qcpv krb5kdc[12387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706945, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:35:45.843333 13157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:45.828699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:59129 (local address 127.28.207.126:36041)
0929 18:35:45.828909 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:45.828913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:45.830149 (+ 1236us) server_negotiation.cc:407] Connection header received
0929 18:35:45.831637 (+ 1488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:45.831641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:45.831706 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:45.831811 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:45.833084 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:45.833655 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:45.834828 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:45.835108 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:45.835222 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:45.838246 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:45.838252 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:45.838257 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:45.838295 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:45.840356 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:45.841146 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:45.841152 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:45.841156 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:45.841268 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:45.841740 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:45.841745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:45.841750 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:45.842050 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:45.842190 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:45.842839 (+ 649us) server_negotiation.cc:299] Negotiation successful
0929 18:35:45.843051 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 18:35:45.844389 13295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:45.829282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.66:59129)
0929 18:35:45.829926 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:45.829984 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:45.831393 (+ 1409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:45.832007 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:45.832014 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:45.832413 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:45.832927 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:45.832940 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:45.833784 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:45.833787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:45.834700 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:45.834709 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:45.835285 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:45.835289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:45.835477 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:45.836168 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:45.836197 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:45.837992 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:45.840538 (+ 2546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:45.840544 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:45.840560 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:45.841000 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:45.841418 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:45.841421 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:45.841422 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:45.841620 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:45.842206 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:45.842212 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:45.842550 (+ 338us) client_negotiation.cc:769] Sending connection context
0929 18:35:45.842746 (+ 196us) client_negotiation.cc:240] Negotiation successful
0929 18:35:45.843176 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":168,"threads_started":1}
I20260929 18:35:45.846032 13293 heartbeater.cc:347] Connected to a master server at 127.28.207.126:36041
I20260929 18:35:45.846462 13293 heartbeater.cc:464] Registering TS with master...
I20260929 18:35:45.847316 13293 heartbeater.cc:511] Master 127.28.207.126:36041 requested a full tablet report, sending...
I20260929 18:35:45.848799 12967 ts_manager.cc:194] Registered new tserver with Master: 12e32b320f0349d18c5c4811cae01a0d (127.28.207.66:46547)
I20260929 18:35:45.849517 12967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.66@KRBTEST.COM'} at 127.28.207.66:59129
W20260929 18:35:45.953493 13296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:35:45.953773 13296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:35:45.953817 13296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:35:45.957864 13296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:35:45.957938 13296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:35:45.957999 13296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.67
I20260929 18:35:45.963312 13296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:42001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.67:35193
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.207.67
--webserver_port=36799
--tserver_master_addrs=127.28.207.126:36041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.67
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:35:45.964419 13296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:35:45.965488 13296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:35:45.975826 13304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:45.975826 13301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:35:45.975867 13302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:35:45.976295 13296 server_base.cc:1034] running on GCE node
I20260929 18:35:45.976719 13296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:35:45.977422 13296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:35:45.978622 13296 hybrid_clock.cc:648] HybridClock initialized: now 1790706945978599 us; error 49 us; skew 500 ppm
Sep 29 18:35:45 dist-test-slave-qcpv krb5kdc[12387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706945, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:35:45.981900 13296 init.cc:377] Logged in from keytab as kudu/127.28.207.67@KRBTEST.COM (short username kudu)
I20260929 18:35:45.982841 13296 webserver.cc:460] Webserver started at http://127.28.207.67:36799/ using document root <none> and password file <none>
I20260929 18:35:45.983453 13296 fs_manager.cc:359] Metadata directory not provided
I20260929 18:35:45.983530 13296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:35:45.987725 13296 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260929 18:35:45.990065 13311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:45.991673 13296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:35:45.991792 13296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/wal
uuid: "74087a8bd66b403d8c9a22b68b090fd7"
format_stamp: "Formatted at 2026-09-29 18:35:24 on dist-test-slave-qcpv"
server_key: "50f25d43b0513d5e9699c43900c84c62"
server_key_iv: "6ec32ed61e9960a8905b66409d1baf67"
server_key_version: "encryptionkey@0"
I20260929 18:35:45.992409 13296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:35:46.008107 13296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:35:46.011767 13296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:35:46.011977 13296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:35:46.012565 13296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:35:46.013504 13296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:35:46.013559 13296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:46.013610 13296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:35:46.013675 13296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:35:46.026319 13296 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.67:35193
I20260929 18:35:46.026425 13424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.67:35193 every 8 connection(s)
I20260929 18:35:46.027824 13296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
I20260929 18:35:46.034917 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 13296
Sep 29 18:35:46 dist-test-slave-qcpv krb5kdc[12387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706945, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:35:46.044992 13157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:46.030408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:41173 (local address 127.28.207.126:36041)
0929 18:35:46.030634 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:46.030640 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:46.031928 (+ 1288us) server_negotiation.cc:407] Connection header received
0929 18:35:46.032976 (+ 1048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:46.032979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:46.033035 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:46.033130 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:46.034677 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.035435 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.037068 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.037447 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.037577 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:46.040761 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:46.040768 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:46.040777 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:46.040805 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:46.042534 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:46.043149 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:46.043154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:46.043177 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:46.043287 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:46.043709 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:46.043711 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:46.043713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:46.043900 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:46.043979 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:46.044676 (+ 697us) server_negotiation.cc:299] Negotiation successful
0929 18:35:46.044828 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 18:35:46.045948 13429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:46.035886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59314 (local address 127.28.207.126:36041)
0929 18:35:46.036180 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:46.036184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:46.036373 (+ 189us) server_negotiation.cc:407] Connection header received
0929 18:35:46.036539 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:46.036542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:46.036590 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:46.036713 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:46.037632 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.038419 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.039503 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.039776 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.039900 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:46.041814 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:46.041821 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:46.041826 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:46.041889 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:46.043926 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:46.044365 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:46.044368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:46.044370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:46.044458 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:46.044829 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:46.044832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:46.044834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:46.045227 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:46.045462 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:46.045643 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 18:35:46.045778 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":111,"threads_started":1}
I20260929 18:35:46.046393 13427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:46.030905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.67:41173)
0929 18:35:46.031723 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:46.031776 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:46.032732 (+ 956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:46.033356 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:46.033367 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:46.033806 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:35:46.034454 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:46.034470 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.035587 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.035593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:46.036934 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:46.036944 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.037682 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.037686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:46.037904 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:46.038681 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:35:46.038700 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:35:46.040471 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:35:46.042656 (+ 2185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:46.042663 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:46.042685 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:46.043044 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:46.043389 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:35:46.043395 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:35:46.043398 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:35:46.043594 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:35:46.044025 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:35:46.044036 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:35:46.044359 (+ 323us) client_negotiation.cc:769] Sending connection context
0929 18:35:46.044590 (+ 231us) client_negotiation.cc:240] Negotiation successful
0929 18:35:46.045010 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":128,"threads_started":1}
I20260929 18:35:46.048008 13425 heartbeater.cc:347] Connected to a master server at 127.28.207.126:36041
I20260929 18:35:46.048311 13425 heartbeater.cc:464] Registering TS with master...
I20260929 18:35:46.048929 13425 heartbeater.cc:511] Master 127.28.207.126:36041 requested a full tablet report, sending...
I20260929 18:35:46.050113 12967 ts_manager.cc:194] Registered new tserver with Master: 74087a8bd66b403d8c9a22b68b090fd7 (127.28.207.67:35193)
I20260929 18:35:46.050729 12967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.67@KRBTEST.COM'} at 127.28.207.67:41173
I20260929 18:35:46.060982 29501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:35:46.065874 13429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:46.061783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59328 (local address 127.28.207.126:36041)
0929 18:35:46.061968 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:46.061978 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:46.062227 (+ 249us) server_negotiation.cc:407] Connection header received
0929 18:35:46.062333 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:46.062336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:46.062467 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:46.062601 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:35:46.062922 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.063614 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.064556 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.064808 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.064907 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:46.065229 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:35:46.065425 (+ 196us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 18:35:46.065700 (+ 275us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59328: token expired
Metrics: {"server-negotiator.queue_time_us":73}
W20260929 18:35:46.065969 13429 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59328: token expired
W20260929 18:35:46.066418 12895 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.28.207.126:36041)
I20260929 18:35:46.074518 13429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:46.066813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59332 (local address 127.28.207.126:36041)
0929 18:35:46.066999 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:46.067004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:46.067018 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:35:46.067135 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:46.067137 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:46.067176 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:46.067257 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:35:46.067924 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.068444 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.069380 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.069603 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.069705 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:46.071284 (+ 1579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:35:46.071288 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:35:46.071290 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:35:46.071313 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:35:46.072764 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:46.073221 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:46.073225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:46.073227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:46.073311 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:35:46.073580 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:35:46.073585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:35:46.073587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:35:46.073784 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:35:46.073927 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:46.074180 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 18:35:46.074311 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 18:35:46.077543 12967 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59332:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:35:46.081264 12967 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:35:46.091153 13089 tablet_service.cc:1467] Processing CreateTablet for tablet 8baead015be7469888fee9b504a00ea4 (DEFAULT_TABLE table=test-table [id=adce81a7ff774eceaf9b62405254111c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:46.092025 13089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8baead015be7469888fee9b504a00ea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:46.095818 13162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:46.089668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:46547 (local address 127.0.0.1:40948)
0929 18:35:46.089813 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:46.089833 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:46.089923 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:46.090818 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:46.090822 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:46.090850 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:46.090996 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:46.091005 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.092746 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.092752 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:46.094032 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:46.094041 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.095217 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.095221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:46.095355 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:46.095437 (+ 82us) client_negotiation.cc:769] Sending connection context
0929 18:35:46.095519 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 18:35:46.095628 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":3}
I20260929 18:35:46.095968 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:46.089900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40948 (local address 127.28.207.66:46547)
0929 18:35:46.090257 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:46.090270 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:46.090296 (+ 26us) server_negotiation.cc:407] Connection header received
0929 18:35:46.090383 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:46.090389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:46.090633 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:46.091175 (+ 542us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:46.091685 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.092633 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.094169 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.095083 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.095196 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:46.095353 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:46.095672 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 18:35:46.095793 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":149,"threads_started":1}
I20260929 18:35:46.096072 13047 tablet_bootstrap.cc:492] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594: Bootstrap starting.
I20260929 18:35:46.096561 13436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:46.090297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55516 (local address 127.28.207.67:35193)
0929 18:35:46.090816 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:46.090829 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:46.090877 (+ 48us) server_negotiation.cc:407] Connection header received
0929 18:35:46.090980 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:46.090985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:46.091394 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:46.091708 (+ 314us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:46.091967 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.093043 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.094720 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.095686 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.095797 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:46.095939 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:46.096308 (+ 369us) server_negotiation.cc:299] Negotiation successful
0929 18:35:46.096415 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":166,"threads_started":1}
I20260929 18:35:46.096589 13435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:46.089936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:35193 (local address 127.0.0.1:55516)
0929 18:35:46.090595 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:46.090613 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:46.090720 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:46.091626 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:46.091629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:46.091646 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:46.091836 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:46.091844 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.093170 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.093173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:46.094553 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:46.094565 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.096013 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.096017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:46.096140 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:46.096170 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 18:35:46.096279 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 18:35:46.096370 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"mutex_wait_us":62,"thread_start_us":154,"threads_started":1}
I20260929 18:35:46.097193 13047 tablet_bootstrap.cc:654] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:46.098825 13047 tablet_bootstrap.cc:492] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594: No bootstrap required, opened a new log
I20260929 18:35:46.098953 13047 ts_tablet_manager.cc:1397] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260929 18:35:46.099452 13047 raft_consensus.cc:348] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e32b320f0349d18c5c4811cae01a0d" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46547 } } peers { permanent_uuid: "74087a8bd66b403d8c9a22b68b090fd7" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 35193 } } peers { permanent_uuid: "9dfe7e3bf3674eed89c366f59653a594" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46723 } }
I20260929 18:35:46.099571 13047 raft_consensus.cc:374] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:46.099606 13047 raft_consensus.cc:729] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dfe7e3bf3674eed89c366f59653a594, State: Initialized, Role: FOLLOWER
I20260929 18:35:46.099748 13047 consensus_queue.cc:260] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e32b320f0349d18c5c4811cae01a0d" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46547 } } peers { permanent_uuid: "74087a8bd66b403d8c9a22b68b090fd7" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 35193 } } peers { permanent_uuid: "9dfe7e3bf3674eed89c366f59653a594" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46723 } }
I20260929 18:35:46.100011 13047 ts_tablet_manager.cc:1428] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 18:35:46.099931 13227 tablet_service.cc:1467] Processing CreateTablet for tablet 8baead015be7469888fee9b504a00ea4 (DEFAULT_TABLE table=test-table [id=adce81a7ff774eceaf9b62405254111c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:46.100376 13359 tablet_service.cc:1467] Processing CreateTablet for tablet 8baead015be7469888fee9b504a00ea4 (DEFAULT_TABLE table=test-table [id=adce81a7ff774eceaf9b62405254111c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:35:46.101121 13227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8baead015be7469888fee9b504a00ea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:46.101524 13359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8baead015be7469888fee9b504a00ea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:35:46.108930 13438 tablet_bootstrap.cc:492] T 8baead015be7469888fee9b504a00ea4 P 12e32b320f0349d18c5c4811cae01a0d: Bootstrap starting.
I20260929 18:35:46.109320 13439 tablet_bootstrap.cc:492] T 8baead015be7469888fee9b504a00ea4 P 74087a8bd66b403d8c9a22b68b090fd7: Bootstrap starting.
I20260929 18:35:46.111806 13438 tablet_bootstrap.cc:654] T 8baead015be7469888fee9b504a00ea4 P 12e32b320f0349d18c5c4811cae01a0d: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:46.111920 13439 tablet_bootstrap.cc:654] T 8baead015be7469888fee9b504a00ea4 P 74087a8bd66b403d8c9a22b68b090fd7: Neither blocks nor log segments found. Creating new log.
I20260929 18:35:46.112794 13438 log.cc:824] T 8baead015be7469888fee9b504a00ea4 P 12e32b320f0349d18c5c4811cae01a0d: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:46.112813 13439 log.cc:824] T 8baead015be7469888fee9b504a00ea4 P 74087a8bd66b403d8c9a22b68b090fd7: Log is configured to *not* fsync() on all Append() calls
I20260929 18:35:46.115137 13438 tablet_bootstrap.cc:492] T 8baead015be7469888fee9b504a00ea4 P 12e32b320f0349d18c5c4811cae01a0d: No bootstrap required, opened a new log
I20260929 18:35:46.115387 13438 ts_tablet_manager.cc:1397] T 8baead015be7469888fee9b504a00ea4 P 12e32b320f0349d18c5c4811cae01a0d: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 18:35:46.116129 13439 tablet_bootstrap.cc:492] T 8baead015be7469888fee9b504a00ea4 P 74087a8bd66b403d8c9a22b68b090fd7: No bootstrap required, opened a new log
I20260929 18:35:46.116315 13439 ts_tablet_manager.cc:1397] T 8baead015be7469888fee9b504a00ea4 P 74087a8bd66b403d8c9a22b68b090fd7: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 18:35:46.119541 13438 raft_consensus.cc:348] T 8baead015be7469888fee9b504a00ea4 P 12e32b320f0349d18c5c4811cae01a0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e32b320f0349d18c5c4811cae01a0d" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46547 } } peers { permanent_uuid: "74087a8bd66b403d8c9a22b68b090fd7" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 35193 } } peers { permanent_uuid: "9dfe7e3bf3674eed89c366f59653a594" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46723 } }
I20260929 18:35:46.119874 13438 raft_consensus.cc:374] T 8baead015be7469888fee9b504a00ea4 P 12e32b320f0349d18c5c4811cae01a0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:46.119940 13438 raft_consensus.cc:729] T 8baead015be7469888fee9b504a00ea4 P 12e32b320f0349d18c5c4811cae01a0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e32b320f0349d18c5c4811cae01a0d, State: Initialized, Role: FOLLOWER
I20260929 18:35:46.120074 13439 raft_consensus.cc:348] T 8baead015be7469888fee9b504a00ea4 P 74087a8bd66b403d8c9a22b68b090fd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e32b320f0349d18c5c4811cae01a0d" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46547 } } peers { permanent_uuid: "74087a8bd66b403d8c9a22b68b090fd7" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 35193 } } peers { permanent_uuid: "9dfe7e3bf3674eed89c366f59653a594" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46723 } }
I20260929 18:35:46.120308 13439 raft_consensus.cc:374] T 8baead015be7469888fee9b504a00ea4 P 74087a8bd66b403d8c9a22b68b090fd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:35:46.120383 13439 raft_consensus.cc:729] T 8baead015be7469888fee9b504a00ea4 P 74087a8bd66b403d8c9a22b68b090fd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74087a8bd66b403d8c9a22b68b090fd7, State: Initialized, Role: FOLLOWER
I20260929 18:35:46.120434 13438 consensus_queue.cc:260] T 8baead015be7469888fee9b504a00ea4 P 12e32b320f0349d18c5c4811cae01a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e32b320f0349d18c5c4811cae01a0d" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46547 } } peers { permanent_uuid: "74087a8bd66b403d8c9a22b68b090fd7" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 35193 } } peers { permanent_uuid: "9dfe7e3bf3674eed89c366f59653a594" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46723 } }
I20260929 18:35:46.120786 13439 consensus_queue.cc:260] T 8baead015be7469888fee9b504a00ea4 P 74087a8bd66b403d8c9a22b68b090fd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e32b320f0349d18c5c4811cae01a0d" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46547 } } peers { permanent_uuid: "74087a8bd66b403d8c9a22b68b090fd7" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 35193 } } peers { permanent_uuid: "9dfe7e3bf3674eed89c366f59653a594" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46723 } }
I20260929 18:35:46.121440 13425 heartbeater.cc:503] Master 127.28.207.126:36041 was elected leader, sending a full tablet report...
I20260929 18:35:46.121418 13293 heartbeater.cc:503] Master 127.28.207.126:36041 was elected leader, sending a full tablet report...
I20260929 18:35:46.121539 13438 ts_tablet_manager.cc:1428] T 8baead015be7469888fee9b504a00ea4 P 12e32b320f0349d18c5c4811cae01a0d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 18:35:46.121591 13439 ts_tablet_manager.cc:1428] T 8baead015be7469888fee9b504a00ea4 P 74087a8bd66b403d8c9a22b68b090fd7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260929 18:35:46.162511 13161 raft_consensus.cc:482] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:35:46.162818 13161 raft_consensus.cc:504] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e32b320f0349d18c5c4811cae01a0d" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46547 } } peers { permanent_uuid: "74087a8bd66b403d8c9a22b68b090fd7" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 35193 } } peers { permanent_uuid: "9dfe7e3bf3674eed89c366f59653a594" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46723 } }
I20260929 18:35:46.163995 13161 leader_election.cc:288] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12e32b320f0349d18c5c4811cae01a0d (127.28.207.66:46547), 74087a8bd66b403d8c9a22b68b090fd7 (127.28.207.67:35193)
I20260929 18:35:46.168650 13444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:46.164475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:46547 (local address 127.28.207.65:51903)
0929 18:35:46.164804 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:46.164825 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:46.164997 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:46.165490 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:46.165492 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:46.165507 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:46.165628 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:46.165633 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.166400 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.166406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:46.167445 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:46.167455 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.168128 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.168131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:46.168237 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:46.168287 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 18:35:46.168375 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 18:35:46.168467 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":48,"thread_start_us":118,"threads_started":1}
I20260929 18:35:46.168635 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:46.164582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:51903 (local address 127.28.207.66:46547)
0929 18:35:46.164716 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:46.164720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:46.164922 (+ 202us) server_negotiation.cc:407] Connection header received
0929 18:35:46.165237 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:46.165242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:46.165300 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:46.165347 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:46.165743 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.166290 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.167580 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.167998 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.168128 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:46.168196 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:46.168391 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 18:35:46.168474 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:35:46.168681 13436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:46.165082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:36953 (local address 127.28.207.67:35193)
0929 18:35:46.165210 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:46.165214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:46.165387 (+ 173us) server_negotiation.cc:407] Connection header received
0929 18:35:46.165563 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:46.165566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:46.165616 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:46.165699 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:35:46.165957 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.166487 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.167666 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.168084 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.168212 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:46.168267 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:46.168464 (+ 197us) server_negotiation.cc:299] Negotiation successful
0929 18:35:46.168545 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:35:46.169343 13445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:46.164916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:35193 (local address 127.28.207.65:36953)
0929 18:35:46.165285 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:46.165300 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:35:46.165453 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:35:46.165722 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:35:46.165725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:35:46.165743 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:35:46.165860 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:46.165865 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.166593 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.166599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:46.167543 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:35:46.167552 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.168215 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.168219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:35:46.168312 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:46.168343 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 18:35:46.169027 (+ 684us) client_negotiation.cc:240] Negotiation successful
0929 18:35:46.169125 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":108,"threads_started":1}
I20260929 18:35:46.169564 13247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8baead015be7469888fee9b504a00ea4" candidate_uuid: "9dfe7e3bf3674eed89c366f59653a594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e32b320f0349d18c5c4811cae01a0d" is_pre_election: true
I20260929 18:35:46.169911 13247 raft_consensus.cc:2454] T 8baead015be7469888fee9b504a00ea4 P 12e32b320f0349d18c5c4811cae01a0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dfe7e3bf3674eed89c366f59653a594 in term 0.
I20260929 18:35:46.170254 13379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8baead015be7469888fee9b504a00ea4" candidate_uuid: "9dfe7e3bf3674eed89c366f59653a594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74087a8bd66b403d8c9a22b68b090fd7" is_pre_election: true
I20260929 18:35:46.170440 13041 leader_election.cc:302] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e32b320f0349d18c5c4811cae01a0d, 9dfe7e3bf3674eed89c366f59653a594; no voters:
I20260929 18:35:46.170648 13379 raft_consensus.cc:2454] T 8baead015be7469888fee9b504a00ea4 P 74087a8bd66b403d8c9a22b68b090fd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dfe7e3bf3674eed89c366f59653a594 in term 0.
I20260929 18:35:46.170742 13161 raft_consensus.cc:2781] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:35:46.170878 13161 raft_consensus.cc:482] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:35:46.170950 13161 raft_consensus.cc:3037] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:46.172081 13161 raft_consensus.cc:504] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e32b320f0349d18c5c4811cae01a0d" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46547 } } peers { permanent_uuid: "74087a8bd66b403d8c9a22b68b090fd7" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 35193 } } peers { permanent_uuid: "9dfe7e3bf3674eed89c366f59653a594" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46723 } }
I20260929 18:35:46.172415 13161 leader_election.cc:288] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594 [CANDIDATE]: Term 1 election: Requested vote from peers 12e32b320f0349d18c5c4811cae01a0d (127.28.207.66:46547), 74087a8bd66b403d8c9a22b68b090fd7 (127.28.207.67:35193)
I20260929 18:35:46.172883 13247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8baead015be7469888fee9b504a00ea4" candidate_uuid: "9dfe7e3bf3674eed89c366f59653a594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12e32b320f0349d18c5c4811cae01a0d"
I20260929 18:35:46.172868 13379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8baead015be7469888fee9b504a00ea4" candidate_uuid: "9dfe7e3bf3674eed89c366f59653a594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74087a8bd66b403d8c9a22b68b090fd7"
I20260929 18:35:46.173022 13247 raft_consensus.cc:3037] T 8baead015be7469888fee9b504a00ea4 P 12e32b320f0349d18c5c4811cae01a0d [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:46.173033 13379 raft_consensus.cc:3037] T 8baead015be7469888fee9b504a00ea4 P 74087a8bd66b403d8c9a22b68b090fd7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:35:46.174083 13247 raft_consensus.cc:2454] T 8baead015be7469888fee9b504a00ea4 P 12e32b320f0349d18c5c4811cae01a0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dfe7e3bf3674eed89c366f59653a594 in term 1.
I20260929 18:35:46.174413 13041 leader_election.cc:302] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12e32b320f0349d18c5c4811cae01a0d, 9dfe7e3bf3674eed89c366f59653a594; no voters:
I20260929 18:35:46.174576 13161 raft_consensus.cc:2781] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:35:46.174634 13379 raft_consensus.cc:2454] T 8baead015be7469888fee9b504a00ea4 P 74087a8bd66b403d8c9a22b68b090fd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dfe7e3bf3674eed89c366f59653a594 in term 1.
I20260929 18:35:46.174787 13161 raft_consensus.cc:686] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594 [term 1 LEADER]: Becoming Leader. State: Replica: 9dfe7e3bf3674eed89c366f59653a594, State: Running, Role: LEADER
I20260929 18:35:46.175099 13161 consensus_queue.cc:237] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e32b320f0349d18c5c4811cae01a0d" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46547 } } peers { permanent_uuid: "74087a8bd66b403d8c9a22b68b090fd7" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 35193 } } peers { permanent_uuid: "9dfe7e3bf3674eed89c366f59653a594" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46723 } }
I20260929 18:35:46.178294 12967 catalog_manager.cc:5495] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dfe7e3bf3674eed89c366f59653a594 (127.28.207.65). New cstate: current_term: 1 leader_uuid: "9dfe7e3bf3674eed89c366f59653a594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e32b320f0349d18c5c4811cae01a0d" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74087a8bd66b403d8c9a22b68b090fd7" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 35193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9dfe7e3bf3674eed89c366f59653a594" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46723 } health_report { overall_health: HEALTHY } } }
I20260929 18:35:46.198479 13449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:46.194454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60394 (local address 127.28.207.65:46723)
0929 18:35:46.194847 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0929 18:35:46.194905 (+ 58us) server_negotiation.cc:399] Waiting for connection header
0929 18:35:46.194928 (+ 23us) server_negotiation.cc:407] Connection header received
0929 18:35:46.195003 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:35:46.195005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:35:46.195061 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:35:46.195179 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:35:46.195457 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.195994 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.196962 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:35:46.197183 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:35:46.197319 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:35:46.197561 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:35:46.197983 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:35:46.198071 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 18:35:46.198238 (+ 167us) server_negotiation.cc:299] Negotiation successful
0929 18:35:46.198285 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":165,"threads_started":1}
I20260929 18:35:46.202940 13379 raft_consensus.cc:1261] T 8baead015be7469888fee9b504a00ea4 P 74087a8bd66b403d8c9a22b68b090fd7 [term 1 FOLLOWER]: Refusing update from remote peer 9dfe7e3bf3674eed89c366f59653a594: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:35:46.203284 13247 raft_consensus.cc:1261] T 8baead015be7469888fee9b504a00ea4 P 12e32b320f0349d18c5c4811cae01a0d [term 1 FOLLOWER]: Refusing update from remote peer 9dfe7e3bf3674eed89c366f59653a594: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:35:46.203720 13161 consensus_queue.cc:1035] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74087a8bd66b403d8c9a22b68b090fd7" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 35193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:35:46.203984 13446 consensus_queue.cc:1035] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e32b320f0349d18c5c4811cae01a0d" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:35:46.212245 13450 mvcc.cc:205] Tried to move back new op lower bound from 7334735651639095296 to 7334735651535867904. Current Snapshot: MvccSnapshot[applied={T|T < 7334735651639095296}]
I20260929 18:35:46.215988 13451 mvcc.cc:205] Tried to move back new op lower bound from 7334735651639095296 to 7334735651535867904. Current Snapshot: MvccSnapshot[applied={T|T < 7334735651639095296}]
I20260929 18:35:46.222348 12966 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:35:46.222558 12966 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:35:46.225845 12966 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46: Sending DeleteTablet for 3 replicas of tablet 8baead015be7469888fee9b504a00ea4
I20260929 18:35:46.226609 13089 tablet_service.cc:1514] Processing DeleteTablet for tablet 8baead015be7469888fee9b504a00ea4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:46 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:60392
I20260929 18:35:46.226752 13359 tablet_service.cc:1514] Processing DeleteTablet for tablet 8baead015be7469888fee9b504a00ea4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:46 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:55516
I20260929 18:35:46.227097 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 13005
I20260929 18:35:46.227064 13227 tablet_service.cc:1514] Processing DeleteTablet for tablet 8baead015be7469888fee9b504a00ea4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:35:46 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:40948
I20260929 18:35:46.227866 13461 tablet_replica.cc:330] T 8baead015be7469888fee9b504a00ea4 P 74087a8bd66b403d8c9a22b68b090fd7: stopping tablet replica
I20260929 18:35:46.228216 13461 raft_consensus.cc:2229] T 8baead015be7469888fee9b504a00ea4 P 74087a8bd66b403d8c9a22b68b090fd7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:35:46.228580 13461 raft_consensus.cc:2258] T 8baead015be7469888fee9b504a00ea4 P 74087a8bd66b403d8c9a22b68b090fd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:35:46.230386 13461 ts_tablet_manager.cc:1905] T 8baead015be7469888fee9b504a00ea4 P 74087a8bd66b403d8c9a22b68b090fd7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:35:46.231617 13462 tablet_replica.cc:330] T 8baead015be7469888fee9b504a00ea4 P 12e32b320f0349d18c5c4811cae01a0d: stopping tablet replica
I20260929 18:35:46.232044 13462 raft_consensus.cc:2229] T 8baead015be7469888fee9b504a00ea4 P 12e32b320f0349d18c5c4811cae01a0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:35:46.232432 13462 raft_consensus.cc:2258] T 8baead015be7469888fee9b504a00ea4 P 12e32b320f0349d18c5c4811cae01a0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:35:46.233345 13461 ts_tablet_manager.cc:1918] T 8baead015be7469888fee9b504a00ea4 P 74087a8bd66b403d8c9a22b68b090fd7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:35:46.233462 13461 log.cc:1196] T 8baead015be7469888fee9b504a00ea4 P 74087a8bd66b403d8c9a22b68b090fd7: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/wal/wals/8baead015be7469888fee9b504a00ea4
I20260929 18:35:46.233808 13461 ts_tablet_manager.cc:1939] T 8baead015be7469888fee9b504a00ea4 P 74087a8bd66b403d8c9a22b68b090fd7: Deleting consensus metadata
I20260929 18:35:46.234277 13462 ts_tablet_manager.cc:1905] T 8baead015be7469888fee9b504a00ea4 P 12e32b320f0349d18c5c4811cae01a0d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 18:35:46.234897 12954 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.207.65:46723 (error 108)
I20260929 18:35:46.235507 12951 catalog_manager.cc:4841] TS 74087a8bd66b403d8c9a22b68b090fd7 (127.28.207.67:35193): tablet 8baead015be7469888fee9b504a00ea4 (table test-table [id=adce81a7ff774eceaf9b62405254111c]) successfully deleted
I20260929 18:35:46.235882 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 13159
I20260929 18:35:46.237944 13435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:46.237024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:46723 (local address 127.0.0.1:60396)
0929 18:35:46.237779 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:46.237858 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.65:46723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":653}
W20260929 18:35:46.238137 12954 catalog_manager.cc:4584] TS 9dfe7e3bf3674eed89c366f59653a594 (127.28.207.65:46723): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8baead015be7469888fee9b504a00ea4: Network error: Client connection negotiation failed: client connection to 127.28.207.65:46723: connect: Connection refused (error 111)
I20260929 18:35:46.244023 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 13296
I20260929 18:35:46.244633 13435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:35:46.244259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:46547 (local address 127.0.0.1:40960)
0929 18:35:46.244451 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 18:35:46.244520 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.66:46547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260929 18:35:46.250764 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 12934
I20260929 18:36:06.259166 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:36041
--webserver_interface=127.28.207.126
--webserver_port=36549
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:42001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:36041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:06.373500 13464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:06.373843 13464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:06.373921 13464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:06.378012 13464 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 18:36:06.378109 13464 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 18:36:06.378144 13464 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 18:36:06.378175 13464 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:36:06.378237 13464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:06.378301 13464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:06.378341 13464 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:36:06.378381 13464 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:36:06.383781 13464 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:42001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:36041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:36041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=36549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:06.384986 13464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:06.386073 13464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:06.395756 13469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:06.395852 13472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:06.395952 13464 server_base.cc:1034] running on GCE node
W20260929 18:36:06.395850 13470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:06.396862 13464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:06.398272 13464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:06.399505 13464 hybrid_clock.cc:648] HybridClock initialized: now 1790706966399454 us; error 60 us; skew 500 ppm
Sep 29 18:36:06 dist-test-slave-qcpv krb5kdc[12387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706966, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:06.403522 13464 init.cc:377] Logged in from keytab as kudu/127.28.207.126@KRBTEST.COM (short username kudu)
I20260929 18:36:06.404841 13464 webserver.cc:460] Webserver started at http://127.28.207.126:36549/ using document root <none> and password file <none>
I20260929 18:36:06.405516 13464 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:06.405587 13464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:06.410956 13464 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:36:06.413913 13479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:06.415879 13464 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:36:06.416067 13464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "30e2464e79e64d83a9d4b044ec3e3a46"
format_stamp: "Formatted at 2026-09-29 18:35:23 on dist-test-slave-qcpv"
server_key: "42857d24f67d8174476fabdc8ede8779"
server_key_iv: "29b71d3de15d499b25bb7a1c75d33bd4"
server_key_version: "encryptionkey@0"
I20260929 18:36:06.416872 13464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:06.439888 13464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:06.444588 13464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:06.444814 13464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:06.454506 13464 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:36041
I20260929 18:36:06.454900 13531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:36041 every 8 connection(s)
I20260929 18:36:06.455991 13464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:36:06.457940 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 13464
I20260929 18:36:06.458618 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.65:46723
--local_ip_for_outbound_sockets=127.28.207.65
--tserver_master_addrs=127.28.207.126:36041
--webserver_port=34605
--webserver_interface=127.28.207.65
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:42001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:36:06.461184 13532 sys_catalog.cc:263] Verifying existing consensus state
I20260929 18:36:06.463981 13532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46: Bootstrap starting.
I20260929 18:36:06.488862 13532 log.cc:824] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:06.503978 13532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:36:06.504522 13532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46: Bootstrap complete.
I20260929 18:36:06.509784 13532 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e2464e79e64d83a9d4b044ec3e3a46" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 36041 } }
I20260929 18:36:06.510327 13532 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30e2464e79e64d83a9d4b044ec3e3a46, State: Initialized, Role: FOLLOWER
I20260929 18:36:06.511091 13532 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e2464e79e64d83a9d4b044ec3e3a46" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 36041 } }
I20260929 18:36:06.511264 13532 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:36:06.511368 13532 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:36:06.511515 13532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [term 2 FOLLOWER]: Advancing to term 3
I20260929 18:36:06.514074 13532 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e2464e79e64d83a9d4b044ec3e3a46" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 36041 } }
I20260929 18:36:06.514506 13532 leader_election.cc:302] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30e2464e79e64d83a9d4b044ec3e3a46; no voters:
I20260929 18:36:06.515033 13532 leader_election.cc:288] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 18:36:06.515265 13536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 18:36:06.515625 13536 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [term 3 LEADER]: Becoming Leader. State: Replica: 30e2464e79e64d83a9d4b044ec3e3a46, State: Running, Role: LEADER
I20260929 18:36:06.516029 13536 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e2464e79e64d83a9d4b044ec3e3a46" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 36041 } }
I20260929 18:36:06.516603 13532 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:36:06.517433 13536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30e2464e79e64d83a9d4b044ec3e3a46. Latest consensus state: current_term: 3 leader_uuid: "30e2464e79e64d83a9d4b044ec3e3a46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e2464e79e64d83a9d4b044ec3e3a46" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 36041 } } }
I20260929 18:36:06.517616 13536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:06.518115 13537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "30e2464e79e64d83a9d4b044ec3e3a46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e2464e79e64d83a9d4b044ec3e3a46" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 36041 } } }
I20260929 18:36:06.518254 13537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:06.518105 13543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:36:06.524099 13543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:36:06.524730 13543 catalog_manager.cc:1259] Loaded cluster ID: b8357524696649b287da1fcb2b016672
I20260929 18:36:06.524924 13543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:36:06.527473 13543 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:36:06.528263 13543 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46: Loaded TSK: 0
I20260929 18:36:06.528359 13543 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 30e2464e79e64d83a9d4b044ec3e3a46: Loaded TSK: 1
F20260929 18:36:06.529173 13543 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f862cf8a9ad google::LogMessage::Fail() at ??:0
@ 0x7f862cf8d88a google::LogMessage::SendToLog() at ??:0
@ 0x7f862cf8a4de google::LogMessage::Flush() at ??:0
@ 0x7f862cf8ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f862fc4e18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f862fc46f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f86378f57bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f86378cf99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f86378d0178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f8637911df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f863793082b std::function<>::operator()() at ??:0
@ 0x7f86378cfb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f86378d0f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f86378cd579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f8637911640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f862f1dc3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f862f1dcc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f862f1de3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f862f1cd512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f86308156db start_thread at ??:0
@ 0x7f862ba8861f clone at ??:0
W20260929 18:36:06.627869 13534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:06.628257 13534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:06.628332 13534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:06.634727 13534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:06.634882 13534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:06.634982 13534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.65
I20260929 18:36:06.643402 13534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:42001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.65:46723
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.207.65
--webserver_port=34605
--tserver_master_addrs=127.28.207.126:36041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.65
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:06.645012 13534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:06.646445 13534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:06.659544 13580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:06.659577 13579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:06.659952 13582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:06.660710 13534 server_base.cc:1034] running on GCE node
I20260929 18:36:06.661221 13534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:06.662068 13534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:06.663281 13534 hybrid_clock.cc:648] HybridClock initialized: now 1790706966663253 us; error 39 us; skew 500 ppm
Sep 29 18:36:06 dist-test-slave-qcpv krb5kdc[12387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706966, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:06.667115 13534 init.cc:377] Logged in from keytab as kudu/127.28.207.65@KRBTEST.COM (short username kudu)
I20260929 18:36:06.668290 13534 webserver.cc:460] Webserver started at http://127.28.207.65:34605/ using document root <none> and password file <none>
I20260929 18:36:06.669008 13534 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:06.669103 13534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:06.674177 13534 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:36:06.677259 13589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:06.679291 13534 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:36:06.679489 13534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/wal
uuid: "9dfe7e3bf3674eed89c366f59653a594"
format_stamp: "Formatted at 2026-09-29 18:35:24 on dist-test-slave-qcpv"
server_key: "344cfc47f7a3d5e65f98bd25d917a8c4"
server_key_iv: "ba61fbd4b04313419210b3d5a5f13ac9"
server_key_version: "encryptionkey@0"
I20260929 18:36:06.680250 13534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:06.708892 13534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:06.713060 13534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:06.713317 13534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:06.714098 13534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:06.715758 13596 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 18:36:06.722036 13534 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 18:36:06.722155 13534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260929 18:36:06.722261 13534 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 18:36:06.724874 13534 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 18:36:06.724960 13534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260929 18:36:06.725529 13596 tablet_bootstrap.cc:492] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594: Bootstrap starting.
I20260929 18:36:06.739389 13703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.65:46723 every 8 connection(s)
I20260929 18:36:06.739358 13534 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.65:46723
I20260929 18:36:06.740833 13534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
I20260929 18:36:06.749065 13596 log.cc:824] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:06.749222 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 13534
I20260929 18:36:06.749727 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.66:46547
--local_ip_for_outbound_sockets=127.28.207.66
--tserver_master_addrs=127.28.207.126:36041
--webserver_port=44385
--webserver_interface=127.28.207.66
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:42001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:36:06.758256 13596 tablet_bootstrap.cc:492] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:36:06.758754 13596 tablet_bootstrap.cc:492] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594: Bootstrap complete.
I20260929 18:36:06.760280 13596 ts_tablet_manager.cc:1397] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594: Time spent bootstrapping tablet: real 0.035s user 0.029s sys 0.004s
I20260929 18:36:06.763651 13596 raft_consensus.cc:348] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e32b320f0349d18c5c4811cae01a0d" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46547 } } peers { permanent_uuid: "74087a8bd66b403d8c9a22b68b090fd7" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 35193 } } peers { permanent_uuid: "9dfe7e3bf3674eed89c366f59653a594" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46723 } }
I20260929 18:36:06.764240 13596 raft_consensus.cc:729] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dfe7e3bf3674eed89c366f59653a594, State: Initialized, Role: FOLLOWER
I20260929 18:36:06.764812 13596 consensus_queue.cc:260] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e32b320f0349d18c5c4811cae01a0d" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46547 } } peers { permanent_uuid: "74087a8bd66b403d8c9a22b68b090fd7" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 35193 } } peers { permanent_uuid: "9dfe7e3bf3674eed89c366f59653a594" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46723 } }
I20260929 18:36:06.766075 13596 ts_tablet_manager.cc:1428] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260929 18:36:06.793143 13706 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:36:06.744069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.65:50605)
0929 18:36:06.747298 (+ 3229us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:06.747361 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:06.748687 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:06.792190 (+ 43503us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":2960,"thread_start_us":158,"threads_started":1}
W20260929 18:36:06.793754 13704 heartbeater.cc:644] Failed to heartbeat to 127.28.207.126:36041 (0 consecutive failures): Network error: Failed to ping master at 127.28.207.126:36041: Client connection negotiation failed: client connection to 127.28.207.126:36041: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 18:36:06.794703 13706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:06.794412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.65:44629)
0929 18:36:06.794572 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:06.794646 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 18:36:06.795745 13706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:06.795307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.65:39225)
0929 18:36:06.795592 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:06.795658 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211}
W20260929 18:36:06.795936 13704 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 18:36:06.883297 13708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:06.883668 13708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:06.883754 13708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:06.888226 13708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:06.888356 13708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:06.888465 13708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.66
I20260929 18:36:06.894585 13708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:42001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.66:46547
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.207.66
--webserver_port=44385
--tserver_master_addrs=127.28.207.126:36041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.66
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:06.896231 13708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:06.897630 13708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:06.909706 13717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:06.909683 13714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:06.909744 13715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:06.909947 13708 server_base.cc:1034] running on GCE node
I20260929 18:36:06.910791 13708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:06.911734 13708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:06.912943 13708 hybrid_clock.cc:648] HybridClock initialized: now 1790706966912915 us; error 73 us; skew 500 ppm
Sep 29 18:36:06 dist-test-slave-qcpv krb5kdc[12387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706966, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:06.916507 13708 init.cc:377] Logged in from keytab as kudu/127.28.207.66@KRBTEST.COM (short username kudu)
I20260929 18:36:06.917586 13708 webserver.cc:460] Webserver started at http://127.28.207.66:44385/ using document root <none> and password file <none>
I20260929 18:36:06.918267 13708 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:06.918339 13708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:06.923297 13708 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:36:06.926204 13724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:06.927963 13708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:36:06.928105 13708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/wal
uuid: "12e32b320f0349d18c5c4811cae01a0d"
format_stamp: "Formatted at 2026-09-29 18:35:24 on dist-test-slave-qcpv"
server_key: "501e0055f9101ec05c9cb1cde80eca59"
server_key_iv: "2c9fd2e5322082e5eaef49d2d004141f"
server_key_version: "encryptionkey@0"
I20260929 18:36:06.928759 13708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:06.949082 13708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:06.953285 13708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:06.953496 13708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:06.954241 13708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:06.955809 13731 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 18:36:06.960446 13708 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 18:36:06.960547 13708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260929 18:36:06.960856 13708 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 18:36:06.963248 13708 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 18:36:06.963305 13708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.003s sys 0.000s
I20260929 18:36:06.963697 13731 tablet_bootstrap.cc:492] T 8baead015be7469888fee9b504a00ea4 P 12e32b320f0349d18c5c4811cae01a0d: Bootstrap starting.
I20260929 18:36:06.969992 13731 log.cc:824] T 8baead015be7469888fee9b504a00ea4 P 12e32b320f0349d18c5c4811cae01a0d: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:06.976224 13708 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.66:46547
I20260929 18:36:06.976298 13839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.66:46547 every 8 connection(s)
I20260929 18:36:06.977305 13708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
I20260929 18:36:06.978273 13731 tablet_bootstrap.cc:492] T 8baead015be7469888fee9b504a00ea4 P 12e32b320f0349d18c5c4811cae01a0d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:36:06.978457 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 13708
I20260929 18:36:06.978763 13731 tablet_bootstrap.cc:492] T 8baead015be7469888fee9b504a00ea4 P 12e32b320f0349d18c5c4811cae01a0d: Bootstrap complete.
I20260929 18:36:06.979027 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.67:35193
--local_ip_for_outbound_sockets=127.28.207.67
--tserver_master_addrs=127.28.207.126:36041
--webserver_port=36799
--webserver_interface=127.28.207.67
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:42001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:36:06.979171 13731 ts_tablet_manager.cc:1397] T 8baead015be7469888fee9b504a00ea4 P 12e32b320f0349d18c5c4811cae01a0d: Time spent bootstrapping tablet: real 0.016s user 0.015s sys 0.000s
I20260929 18:36:06.981305 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:06.979742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.66:47919)
0929 18:36:06.980271 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:06.980415 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":140,"threads_started":1}
W20260929 18:36:06.981985 13840 heartbeater.cc:644] Failed to heartbeat to 127.28.207.126:36041 (0 consecutive failures): Network error: Failed to ping master at 127.28.207.126:36041: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
I20260929 18:36:06.982641 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:06.982415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.66:58883)
0929 18:36:06.982534 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:06.982589 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 18:36:06.983289 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:06.983083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.66:48971)
0929 18:36:06.983186 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:06.983233 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 18:36:06.983464 13840 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 18:36:06.983927 13731 raft_consensus.cc:348] T 8baead015be7469888fee9b504a00ea4 P 12e32b320f0349d18c5c4811cae01a0d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e32b320f0349d18c5c4811cae01a0d" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46547 } } peers { permanent_uuid: "74087a8bd66b403d8c9a22b68b090fd7" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 35193 } } peers { permanent_uuid: "9dfe7e3bf3674eed89c366f59653a594" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46723 } }
I20260929 18:36:06.984673 13731 raft_consensus.cc:729] T 8baead015be7469888fee9b504a00ea4 P 12e32b320f0349d18c5c4811cae01a0d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12e32b320f0349d18c5c4811cae01a0d, State: Initialized, Role: FOLLOWER
I20260929 18:36:06.985283 13731 consensus_queue.cc:260] T 8baead015be7469888fee9b504a00ea4 P 12e32b320f0349d18c5c4811cae01a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e32b320f0349d18c5c4811cae01a0d" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46547 } } peers { permanent_uuid: "74087a8bd66b403d8c9a22b68b090fd7" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 35193 } } peers { permanent_uuid: "9dfe7e3bf3674eed89c366f59653a594" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46723 } }
I20260929 18:36:06.986462 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:06.986230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.66:49201)
0929 18:36:06.986313 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:06.986371 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260929 18:36:06.986531 13731 ts_tablet_manager.cc:1428] T 8baead015be7469888fee9b504a00ea4 P 12e32b320f0349d18c5c4811cae01a0d: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 18:36:06.987187 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:06.986978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.66:40475)
0929 18:36:06.987087 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:06.987134 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 18:36:07.097652 13843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:07.097947 13843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:07.097991 13843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:07.102259 13843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:07.102341 13843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:07.102401 13843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.67
I20260929 18:36:07.107995 13843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:42001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.67:35193
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.207.67
--webserver_port=36799
--tserver_master_addrs=127.28.207.126:36041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.67
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:07.109170 13843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:07.110245 13843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:07.121220 13852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:07.121274 13843 server_base.cc:1034] running on GCE node
W20260929 18:36:07.121248 13849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:07.121227 13850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:07.122234 13843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:07.123102 13843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:07.124310 13843 hybrid_clock.cc:648] HybridClock initialized: now 1790706967124280 us; error 52 us; skew 500 ppm
Sep 29 18:36:07 dist-test-slave-qcpv krb5kdc[12387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706967, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:07.127563 13843 init.cc:377] Logged in from keytab as kudu/127.28.207.67@KRBTEST.COM (short username kudu)
I20260929 18:36:07.128613 13843 webserver.cc:460] Webserver started at http://127.28.207.67:36799/ using document root <none> and password file <none>
I20260929 18:36:07.129266 13843 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:07.129334 13843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:07.134675 13843 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260929 18:36:07.137521 13859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:07.139297 13843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:36:07.139431 13843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/wal
uuid: "74087a8bd66b403d8c9a22b68b090fd7"
format_stamp: "Formatted at 2026-09-29 18:35:24 on dist-test-slave-qcpv"
server_key: "50f25d43b0513d5e9699c43900c84c62"
server_key_iv: "6ec32ed61e9960a8905b66409d1baf67"
server_key_version: "encryptionkey@0"
I20260929 18:36:07.140115 13843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:07.152439 13843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:07.156128 13843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:07.156472 13843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:07.157232 13843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:07.158241 13843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:07.158299 13843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:07.158349 13843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:07.158365 13843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:07.169682 13843 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.67:35193
I20260929 18:36:07.169785 13972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.67:35193 every 8 connection(s)
I20260929 18:36:07.171064 13843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
I20260929 18:36:07.175637 13975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:07.173490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.67:43349)
0929 18:36:07.174100 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:07.174635 (+ 535us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":134,"threads_started":1}
W20260929 18:36:07.176136 13973 heartbeater.cc:644] Failed to heartbeat to 127.28.207.126:36041 (0 consecutive failures): Network error: Failed to ping master at 127.28.207.126:36041: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
I20260929 18:36:07.176859 13975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:07.176603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.67:58789)
0929 18:36:07.176747 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:07.176804 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 18:36:07.177398 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 13843
I20260929 18:36:07.177681 13975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:07.177452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.67:35975)
0929 18:36:07.177589 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:07.177636 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260929 18:36:07.178359 13973 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 18:36:07.797315 13977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:07.796669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.65:50097)
0929 18:36:07.797057 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:07.797157 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":104,"threads_started":1}
I20260929 18:36:07.988523 13978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:07.987981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.66:39039)
0929 18:36:07.988348 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:07.988423 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":99,"threads_started":1}
I20260929 18:36:08.179607 13979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:08.179069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.67:51321)
0929 18:36:08.179412 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:08.179478 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":101,"threads_started":1}
I20260929 18:36:08.230101 13980 raft_consensus.cc:482] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:36:08.230330 13980 raft_consensus.cc:504] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e32b320f0349d18c5c4811cae01a0d" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46547 } } peers { permanent_uuid: "74087a8bd66b403d8c9a22b68b090fd7" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 35193 } } peers { permanent_uuid: "9dfe7e3bf3674eed89c366f59653a594" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46723 } }
I20260929 18:36:08.231544 13980 leader_election.cc:288] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12e32b320f0349d18c5c4811cae01a0d (127.28.207.66:46547), 74087a8bd66b403d8c9a22b68b090fd7 (127.28.207.67:35193)
Sep 29 18:36:08 dist-test-slave-qcpv krb5kdc[12387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706966, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for kudu/127.28.207.66@KRBTEST.COM
Sep 29 18:36:08 dist-test-slave-qcpv krb5kdc[12387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706966, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for kudu/127.28.207.67@KRBTEST.COM
I20260929 18:36:08.255869 13982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:08.232113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:35193 (local address 127.28.207.65:33807)
0929 18:36:08.232670 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:08.232689 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:08.232827 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:08.235065 (+ 2238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:08.235067 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:08.235325 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:08.235498 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:08.235507 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:08.237083 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:08.237086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:08.238035 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:08.238044 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:08.238815 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:08.238819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:08.239041 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:08.246292 (+ 7251us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:08.246318 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:08.248875 (+ 2557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:08.252027 (+ 3152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:08.252035 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:08.252068 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:08.252675 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:08.253138 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:08.253141 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:08.253143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:08.253399 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:08.254396 (+ 997us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:08.254405 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:08.254979 (+ 574us) client_negotiation.cc:769] Sending connection context
0929 18:36:08.255237 (+ 258us) client_negotiation.cc:240] Negotiation successful
0929 18:36:08.255662 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":151,"spinlock_wait_cycles":21120,"thread_start_us":109,"threads_started":1}
I20260929 18:36:08.256145 13983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:08.232520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:33807 (local address 127.28.207.67:35193)
0929 18:36:08.233163 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:08.233200 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:08.233243 (+ 43us) server_negotiation.cc:407] Connection header received
0929 18:36:08.234262 (+ 1019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:08.234269 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:08.234710 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:08.235083 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:08.235754 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:08.236937 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:08.238166 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:08.238669 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:08.238859 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:08.249131 (+ 10272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:08.249141 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:08.249160 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:08.249204 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:08.251837 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:08.252834 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:08.252842 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:08.252848 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:08.253020 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:08.253545 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:08.253548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:08.253549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:08.254213 (+ 664us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:08.254369 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:08.255374 (+ 1005us) server_negotiation.cc:299] Negotiation successful
0929 18:36:08.255746 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":92,"threads_started":1}
I20260929 18:36:08.256464 13981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:08.232042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:38773 (local address 127.28.207.66:46547)
0929 18:36:08.232587 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:08.232607 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:08.232658 (+ 51us) server_negotiation.cc:407] Connection header received
0929 18:36:08.232955 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:08.232965 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:08.233370 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:08.233704 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:08.235514 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:08.236738 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:08.238207 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:08.238670 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:08.238845 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:08.248860 (+ 10015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:08.248869 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:08.248889 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:08.248994 (+ 105us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:08.251837 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:08.252834 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:08.252842 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:08.252848 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:08.253020 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:08.253542 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:08.253546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:08.253548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:08.254213 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:08.254369 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:08.255755 (+ 1386us) server_negotiation.cc:299] Negotiation successful
0929 18:36:08.256249 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":125,"threads_started":1}
I20260929 18:36:08.257366 13977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:08.231838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:46547 (local address 127.28.207.65:38773)
0929 18:36:08.232060 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:08.232076 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:08.232183 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:08.233794 (+ 1611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:08.233802 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:08.234465 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:08.235290 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:08.235313 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:08.236892 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:08.236896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:08.238037 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:08.238047 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:08.238823 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:08.238827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:08.239031 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:08.246261 (+ 7230us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:08.246294 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:08.248540 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:08.252050 (+ 3510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:08.252057 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:08.252067 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:08.252690 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:08.253142 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:08.253145 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:08.253146 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:08.253399 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:08.254396 (+ 997us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:08.254405 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:08.254979 (+ 574us) client_negotiation.cc:769] Sending connection context
0929 18:36:08.255237 (+ 258us) client_negotiation.cc:240] Negotiation successful
0929 18:36:08.255770 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":319,"spinlock_wait_cycles":30720}
I20260929 18:36:08.258415 13794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8baead015be7469888fee9b504a00ea4" candidate_uuid: "9dfe7e3bf3674eed89c366f59653a594" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "12e32b320f0349d18c5c4811cae01a0d" is_pre_election: true
I20260929 18:36:08.258788 13794 raft_consensus.cc:2454] T 8baead015be7469888fee9b504a00ea4 P 12e32b320f0349d18c5c4811cae01a0d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dfe7e3bf3674eed89c366f59653a594 in term 1.
W20260929 18:36:08.259320 13590 leader_election.cc:341] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 74087a8bd66b403d8c9a22b68b090fd7 (127.28.207.67:35193): Not found: Tablet not found: 8baead015be7469888fee9b504a00ea4
I20260929 18:36:08.259598 13590 leader_election.cc:302] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 12e32b320f0349d18c5c4811cae01a0d, 9dfe7e3bf3674eed89c366f59653a594; no voters: 74087a8bd66b403d8c9a22b68b090fd7
I20260929 18:36:08.259920 13980 raft_consensus.cc:2781] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 18:36:08.260040 13980 raft_consensus.cc:482] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:36:08.260100 13980 raft_consensus.cc:3037] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594 [term 1 FOLLOWER]: Advancing to term 2
I20260929 18:36:08.262657 13980 raft_consensus.cc:504] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e32b320f0349d18c5c4811cae01a0d" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46547 } } peers { permanent_uuid: "74087a8bd66b403d8c9a22b68b090fd7" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 35193 } } peers { permanent_uuid: "9dfe7e3bf3674eed89c366f59653a594" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46723 } }
I20260929 18:36:08.263140 13980 leader_election.cc:288] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594 [CANDIDATE]: Term 2 election: Requested vote from peers 12e32b320f0349d18c5c4811cae01a0d (127.28.207.66:46547), 74087a8bd66b403d8c9a22b68b090fd7 (127.28.207.67:35193)
I20260929 18:36:08.263940 13794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8baead015be7469888fee9b504a00ea4" candidate_uuid: "9dfe7e3bf3674eed89c366f59653a594" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "12e32b320f0349d18c5c4811cae01a0d"
I20260929 18:36:08.264150 13794 raft_consensus.cc:3037] T 8baead015be7469888fee9b504a00ea4 P 12e32b320f0349d18c5c4811cae01a0d [term 1 FOLLOWER]: Advancing to term 2
W20260929 18:36:08.264417 13590 leader_election.cc:341] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 74087a8bd66b403d8c9a22b68b090fd7 (127.28.207.67:35193): Not found: Tablet not found: 8baead015be7469888fee9b504a00ea4
I20260929 18:36:08.267074 13794 raft_consensus.cc:2454] T 8baead015be7469888fee9b504a00ea4 P 12e32b320f0349d18c5c4811cae01a0d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dfe7e3bf3674eed89c366f59653a594 in term 2.
I20260929 18:36:08.267565 13590 leader_election.cc:302] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 12e32b320f0349d18c5c4811cae01a0d, 9dfe7e3bf3674eed89c366f59653a594; no voters: 74087a8bd66b403d8c9a22b68b090fd7
I20260929 18:36:08.267810 13980 raft_consensus.cc:2781] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 18:36:08.268026 13980 raft_consensus.cc:686] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594 [term 2 LEADER]: Becoming Leader. State: Replica: 9dfe7e3bf3674eed89c366f59653a594, State: Running, Role: LEADER
I20260929 18:36:08.268411 13980 consensus_queue.cc:237] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e32b320f0349d18c5c4811cae01a0d" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46547 } } peers { permanent_uuid: "74087a8bd66b403d8c9a22b68b090fd7" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 35193 } } peers { permanent_uuid: "9dfe7e3bf3674eed89c366f59653a594" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46723 } }
I20260929 18:36:08.268977 13977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:08.268661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.65:46117)
0929 18:36:08.268847 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:08.268896 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260929 18:36:08.663897 13794 raft_consensus.cc:1261] T 8baead015be7469888fee9b504a00ea4 P 12e32b320f0349d18c5c4811cae01a0d [term 2 FOLLOWER]: Refusing update from remote peer 9dfe7e3bf3674eed89c366f59653a594: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260929 18:36:08.664906 13980 consensus_queue.cc:1035] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12e32b320f0349d18c5c4811cae01a0d" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 18:36:08.667724 13987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:08.667200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.66:54457)
0929 18:36:08.667478 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:08.667584 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":79,"threads_started":1}
I20260929 18:36:08.670595 13977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:08.670285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.65:53715)
0929 18:36:08.670443 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:08.670505 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 18:36:08.670787 13987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:08.670422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.66:38935)
0929 18:36:08.670573 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:08.670662 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 18:36:08.670959 13590 consensus_peers.cc:487] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594 -> Peer 74087a8bd66b403d8c9a22b68b090fd7 (127.28.207.67:35193): Couldn't send request to peer 74087a8bd66b403d8c9a22b68b090fd7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8baead015be7469888fee9b504a00ea4. This is attempt 1: this message will repeat every 5th retry.
I20260929 18:36:09.123728 13991 ts_tablet_manager.cc:927] T 8baead015be7469888fee9b504a00ea4 P 74087a8bd66b403d8c9a22b68b090fd7: Initiating tablet copy from peer 9dfe7e3bf3674eed89c366f59653a594 (127.28.207.65:46723)
I20260929 18:36:09.124409 13991 tablet_copy_client.cc:316] T 8baead015be7469888fee9b504a00ea4 P 74087a8bd66b403d8c9a22b68b090fd7: tablet copy: Beginning tablet copy session from remote peer at address 127.28.207.65:46723
Sep 29 18:36:09 dist-test-slave-qcpv krb5kdc[12387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706967, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for kudu/127.28.207.65@KRBTEST.COM
I20260929 18:36:09.136086 13993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:09.125029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:46723 (local address 127.28.207.67:56869)
0929 18:36:09.125328 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:09.125348 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:09.125466 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:09.126193 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:09.126206 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:09.126622 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:09.126764 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:09.126770 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:09.127900 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:09.127903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:09.128588 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:09.128598 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:09.129357 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:09.129360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:09.129455 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:09.130336 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:09.130347 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:09.131787 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:09.134305 (+ 2518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:09.134314 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:09.134319 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:09.134582 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:09.134946 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:09.134949 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:09.134951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:09.135032 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:09.135502 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:09.135508 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:09.135626 (+ 118us) client_negotiation.cc:769] Sending connection context
0929 18:36:09.135752 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 18:36:09.135916 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":126,"threads_started":1}
I20260929 18:36:09.136193 13994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:09.125213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:56869 (local address 127.28.207.65:46723)
0929 18:36:09.125617 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:09.125627 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:09.125655 (+ 28us) server_negotiation.cc:407] Connection header received
0929 18:36:09.125718 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:09.125724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:09.125951 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:09.126140 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:09.126949 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:09.127769 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:09.128730 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:09.129206 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:09.129353 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:09.131947 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:09.131956 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:09.131971 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:09.131994 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:09.134146 (+ 2152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:09.134697 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:09.134702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:09.134703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:09.134784 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:09.135124 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:09.135126 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:09.135127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:09.135395 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:09.135520 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:09.135861 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 18:36:09.136000 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":97,"threads_started":1}
I20260929 18:36:09.137247 13678 tablet_copy_service.cc:140] P 9dfe7e3bf3674eed89c366f59653a594: Received BeginTabletCopySession request for tablet 8baead015be7469888fee9b504a00ea4 from peer 74087a8bd66b403d8c9a22b68b090fd7 ({username='kudu', principal='kudu/127.28.207.67@KRBTEST.COM'} at 127.28.207.67:56869)
I20260929 18:36:09.137442 13678 tablet_copy_service.cc:161] P 9dfe7e3bf3674eed89c366f59653a594: Beginning new tablet copy session on tablet 8baead015be7469888fee9b504a00ea4 from peer 74087a8bd66b403d8c9a22b68b090fd7 at {username='kudu', principal='kudu/127.28.207.67@KRBTEST.COM'} at 127.28.207.67:56869: session id = 74087a8bd66b403d8c9a22b68b090fd7-8baead015be7469888fee9b504a00ea4
I20260929 18:36:09.138727 13678 tablet_copy_source_session.cc:205] T 8baead015be7469888fee9b504a00ea4 P 9dfe7e3bf3674eed89c366f59653a594: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 18:36:09.143517 13991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8baead015be7469888fee9b504a00ea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:09.150918 13991 tablet_copy_client.cc:791] T 8baead015be7469888fee9b504a00ea4 P 74087a8bd66b403d8c9a22b68b090fd7: tablet copy: Starting download of 0 data blocks...
I20260929 18:36:09.151154 13991 tablet_copy_client.cc:655] T 8baead015be7469888fee9b504a00ea4 P 74087a8bd66b403d8c9a22b68b090fd7: tablet copy: Starting download of 1 WAL segments...
I20260929 18:36:09.153615 13991 tablet_copy_client.cc:531] T 8baead015be7469888fee9b504a00ea4 P 74087a8bd66b403d8c9a22b68b090fd7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 18:36:09.155722 13991 tablet_bootstrap.cc:492] T 8baead015be7469888fee9b504a00ea4 P 74087a8bd66b403d8c9a22b68b090fd7: Bootstrap starting.
I20260929 18:36:09.161742 13991 log.cc:824] T 8baead015be7469888fee9b504a00ea4 P 74087a8bd66b403d8c9a22b68b090fd7: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:09.169401 13991 tablet_bootstrap.cc:492] T 8baead015be7469888fee9b504a00ea4 P 74087a8bd66b403d8c9a22b68b090fd7: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 18:36:09.169866 13991 tablet_bootstrap.cc:492] T 8baead015be7469888fee9b504a00ea4 P 74087a8bd66b403d8c9a22b68b090fd7: Bootstrap complete.
I20260929 18:36:09.170296 13991 ts_tablet_manager.cc:1397] T 8baead015be7469888fee9b504a00ea4 P 74087a8bd66b403d8c9a22b68b090fd7: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260929 18:36:09.173053 13991 raft_consensus.cc:348] T 8baead015be7469888fee9b504a00ea4 P 74087a8bd66b403d8c9a22b68b090fd7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e32b320f0349d18c5c4811cae01a0d" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46547 } } peers { permanent_uuid: "74087a8bd66b403d8c9a22b68b090fd7" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 35193 } } peers { permanent_uuid: "9dfe7e3bf3674eed89c366f59653a594" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46723 } }
I20260929 18:36:09.173362 13991 raft_consensus.cc:729] T 8baead015be7469888fee9b504a00ea4 P 74087a8bd66b403d8c9a22b68b090fd7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74087a8bd66b403d8c9a22b68b090fd7, State: Initialized, Role: FOLLOWER
I20260929 18:36:09.173941 13991 consensus_queue.cc:260] T 8baead015be7469888fee9b504a00ea4 P 74087a8bd66b403d8c9a22b68b090fd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12e32b320f0349d18c5c4811cae01a0d" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 46547 } } peers { permanent_uuid: "74087a8bd66b403d8c9a22b68b090fd7" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 35193 } } peers { permanent_uuid: "9dfe7e3bf3674eed89c366f59653a594" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46723 } }
I20260929 18:36:09.175117 13991 ts_tablet_manager.cc:1428] T 8baead015be7469888fee9b504a00ea4 P 74087a8bd66b403d8c9a22b68b090fd7: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260929 18:36:09.175410 13993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:09.175119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.67:36853)
0929 18:36:09.175263 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:09.175322 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 18:36:09.175901 13678 tablet_copy_service.cc:342] P 9dfe7e3bf3674eed89c366f59653a594: Request end of tablet copy session 74087a8bd66b403d8c9a22b68b090fd7-8baead015be7469888fee9b504a00ea4 received from {username='kudu', principal='kudu/127.28.207.67@KRBTEST.COM'} at 127.28.207.67:56869
I20260929 18:36:09.176048 13678 tablet_copy_service.cc:433] P 9dfe7e3bf3674eed89c366f59653a594: ending tablet copy session 74087a8bd66b403d8c9a22b68b090fd7-8baead015be7469888fee9b504a00ea4 on tablet 8baead015be7469888fee9b504a00ea4 with peer 74087a8bd66b403d8c9a22b68b090fd7
I20260929 18:36:09.176272 13993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:09.175958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.67:59395)
0929 18:36:09.176108 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:09.176175 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 18:36:09.521320 13999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:09.520816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.65:44959)
0929 18:36:09.521121 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:09.521187 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":78,"threads_started":1}
I20260929 18:36:09.521543 13927 raft_consensus.cc:1203] T 8baead015be7469888fee9b504a00ea4 P 74087a8bd66b403d8c9a22b68b090fd7 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260929 18:36:09.522718 13993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:09.522408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.67:60317)
0929 18:36:09.522560 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:09.522632 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 18:36:09.672184 14000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:09.671670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.66:51765)
0929 18:36:09.671971 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:09.672058 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":72,"threads_started":1}
I20260929 18:36:10.523161 14001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:10.522446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.65:46309)
0929 18:36:10.522917 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:10.523016 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":94,"threads_started":1}
I20260929 18:36:10.524314 14002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:10.523736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.67:55453)
0929 18:36:10.524075 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:10.524158 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":85,"threads_started":1}
I20260929 18:36:10.673720 14003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:10.673151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.66:44111)
0929 18:36:10.673537 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:10.673608 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":96,"threads_started":1}
I20260929 18:36:11.525070 14004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:11.524176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.65:38927)
0929 18:36:11.524819 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:11.524908 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":108,"threads_started":1}
I20260929 18:36:11.525672 14005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:11.525145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.67:37933)
0929 18:36:11.525527 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:11.525585 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":110,"threads_started":1}
I20260929 18:36:11.675457 14006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:11.674781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.66:56415)
0929 18:36:11.675230 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:11.675307 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":182,"threads_started":1}
I20260929 18:36:12.527042 14007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:12.526392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.65:54109)
0929 18:36:12.526794 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:12.526938 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":122,"threads_started":1}
I20260929 18:36:12.527372 14008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:12.526808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.67:58227)
0929 18:36:12.527143 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:12.527240 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":143,"threads_started":1}
I20260929 18:36:12.527843 14007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:12.527612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.65:51531)
0929 18:36:12.527742 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:12.527792 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 18:36:12.677727 14009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:12.676907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.66:45409)
0929 18:36:12.677434 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:12.677560 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":106,"threads_started":1}
I20260929 18:36:13.529002 14010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:13.528401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.67:55269)
0929 18:36:13.528748 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:13.528839 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":93,"threads_started":1}
I20260929 18:36:13.529278 14011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:13.528720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.65:60979)
0929 18:36:13.529048 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:13.529117 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":106,"threads_started":1}
I20260929 18:36:13.679248 14012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:13.678662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.66:40875)
0929 18:36:13.679032 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:13.679099 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":95,"threads_started":1}
I20260929 18:36:14.530646 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:14.530093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.65:37895)
0929 18:36:14.530423 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:14.530496 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":117,"threads_started":1}
I20260929 18:36:14.530691 14014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:14.530097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.67:35575)
0929 18:36:14.530527 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:14.530586 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":123,"threads_started":1}
I20260929 18:36:14.681039 14015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:14.680327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.66:53947)
0929 18:36:14.680709 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:14.680896 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":107,"threads_started":1}
I20260929 18:36:15.532320 14017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:15.531717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.65:37711)
0929 18:36:15.532105 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:15.532198 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":134,"threads_started":1}
I20260929 18:36:15.532465 14018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:15.531887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.67:56789)
0929 18:36:15.532234 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:15.532309 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":98,"threads_started":1}
I20260929 18:36:15.682760 14019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:15.682146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.66:51395)
0929 18:36:15.682559 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:15.682647 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":109,"threads_started":1}
I20260929 18:36:16.533933 14020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:16.533408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.67:55597)
0929 18:36:16.533695 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:16.533810 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":112,"threads_started":1}
I20260929 18:36:16.534014 14021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:16.533453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.65:39123)
0929 18:36:16.533820 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:16.533900 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":105,"threads_started":1}
I20260929 18:36:16.684227 14022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:16.683700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.66:47931)
0929 18:36:16.684018 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:16.684090 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":100,"threads_started":1}
I20260929 18:36:17.535527 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:17.535072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.65:59543)
0929 18:36:17.535331 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:17.535401 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":98,"threads_started":1}
I20260929 18:36:17.535699 14023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:17.535027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.67:38765)
0929 18:36:17.535344 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:17.535431 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":117,"threads_started":1}
I20260929 18:36:17.686080 14025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:17.685522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.66:57421)
0929 18:36:17.685862 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:17.685949 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":79,"threads_started":1}
I20260929 18:36:18.537334 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:18.536618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.65:55123)
0929 18:36:18.537076 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:18.537169 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":83,"threads_started":1}
I20260929 18:36:18.537672 14027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:18.536929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.67:36755)
0929 18:36:18.537347 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:18.537423 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":128,"threads_started":1}
I20260929 18:36:18.687698 14028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:18.687162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.66:37453)
0929 18:36:18.687503 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:18.687585 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":99,"threads_started":1}
I20260929 18:36:19.538913 14029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:19.538233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.65:50419)
0929 18:36:19.538627 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:19.538711 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":140,"threads_started":1}
I20260929 18:36:19.539031 14030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:19.538527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.67:33625)
0929 18:36:19.538813 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:19.538915 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":91,"threads_started":1}
I20260929 18:36:19.689469 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:19.688849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.66:60325)
0929 18:36:19.689244 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:19.689345 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":116,"threads_started":1}
I20260929 18:36:20.540400 14032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:20.539836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.67:36933)
0929 18:36:20.540172 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:20.540257 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":78,"threads_started":1}
I20260929 18:36:20.540582 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:20.539906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.65:56445)
0929 18:36:20.540308 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:20.540412 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":126,"threads_started":1}
I20260929 18:36:20.691143 14034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:20.690509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.66:33481)
0929 18:36:20.690942 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:20.691036 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":107,"threads_started":1}
I20260929 18:36:21.542044 14035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:21.541409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.67:47301)
0929 18:36:21.541785 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:21.541892 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":83,"threads_started":1}
I20260929 18:36:21.542270 14036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:21.541678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.65:57117)
0929 18:36:21.542040 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:21.542110 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":109,"threads_started":1}
I20260929 18:36:21.692662 14037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:21.692090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:36041 (local address 127.28.207.66:51395)
0929 18:36:21.692447 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:21.692533 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.126:36041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":77,"threads_started":1}
I20260929 18:36:22.186640 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 13534
I20260929 18:36:22.194124 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 13708
I20260929 18:36:22.201184 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 13843
2026-09-29T18:36:22Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59671 ms)
[----------] 30 tests from SecurityITest (263934 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:36:22 dist-test-slave-qcpv krb5kdc[14040](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:36:22 dist-test-slave-qcpv krb5kdc[14040](info): set up 2 sockets
Sep 29 18:36:22 dist-test-slave-qcpv krb5kdc[14040](info): commencing operation
krb5kdc: starting...
W20260929 18:36:23.275118 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:36:23 dist-test-slave-qcpv krb5kdc[14040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:36:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:36:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.207.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
I20260929 18:36:23.415701 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:45177
--webserver_interface=127.28.207.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:34321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:45177
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:23.549916 14053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:23.550271 14053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:23.550316 14053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:23.554949 14053 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:36:23.555043 14053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:23.555064 14053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:23.555084 14053 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:36:23.555100 14053 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:36:23.561655 14053 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:34321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:45177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:45177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:23.563251 14053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:23.564556 14053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:23.575170 14058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:23.575198 14061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:23.575174 14059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:23.575290 14053 server_base.cc:1034] running on GCE node
I20260929 18:36:23.576260 14053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:23.577641 14053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:23.578951 14053 hybrid_clock.cc:648] HybridClock initialized: now 1790706983578910 us; error 55 us; skew 500 ppm
Sep 29 18:36:23 dist-test-slave-qcpv krb5kdc[14040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706983, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:23.582227 14053 init.cc:377] Logged in from keytab as kudu/127.28.207.126@KRBTEST.COM (short username kudu)
I20260929 18:36:23.583413 14053 webserver.cc:460] Webserver started at http://127.28.207.126:44591/ using document root <none> and password file <none>
I20260929 18:36:23.584076 14053 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:23.584141 14053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:23.584391 14053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:23.587260 14053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "e3a15cc1414145709f47276e5f3b2211"
format_stamp: "Formatted at 2026-09-29 18:36:23 on dist-test-slave-qcpv"
server_key: "2ef625627cd952a86ca8f3ca4e8150d0"
server_key_iv: "32e654e0e9834bdf45c08d66697d5d0f"
server_key_version: "encryptionkey@0"
I20260929 18:36:23.587968 14053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "e3a15cc1414145709f47276e5f3b2211"
format_stamp: "Formatted at 2026-09-29 18:36:23 on dist-test-slave-qcpv"
server_key: "2ef625627cd952a86ca8f3ca4e8150d0"
server_key_iv: "32e654e0e9834bdf45c08d66697d5d0f"
server_key_version: "encryptionkey@0"
I20260929 18:36:23.592880 14053 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 18:36:23.596091 14068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:23.597798 14053 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:36:23.597940 14053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "e3a15cc1414145709f47276e5f3b2211"
format_stamp: "Formatted at 2026-09-29 18:36:23 on dist-test-slave-qcpv"
server_key: "2ef625627cd952a86ca8f3ca4e8150d0"
server_key_iv: "32e654e0e9834bdf45c08d66697d5d0f"
server_key_version: "encryptionkey@0"
I20260929 18:36:23.598078 14053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:23.616664 14053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:23.620425 14053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:23.620622 14053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:23.630080 14053 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:45177
I20260929 18:36:23.630038 14120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:45177 every 8 connection(s)
I20260929 18:36:23.631536 14053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:36:23.634012 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 14053
I20260929 18:36:23.634145 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:36:23.634536 29501 external_mini_cluster.cc:1428] Setting key 04dc0f4856f378824682d9e064ab7afa
I20260929 18:36:23.635228 14121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:23.642333 14121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3a15cc1414145709f47276e5f3b2211: Bootstrap starting.
Sep 29 18:36:23 dist-test-slave-qcpv krb5kdc[14040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:36:23.644922 14121 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3a15cc1414145709f47276e5f3b2211: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:23.645740 14121 log.cc:824] T 00000000000000000000000000000000 P e3a15cc1414145709f47276e5f3b2211: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:23.648435 14121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3a15cc1414145709f47276e5f3b2211: No bootstrap required, opened a new log
I20260929 18:36:23.651321 14124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:23.636373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47880 (local address 127.28.207.126:45177)
0929 18:36:23.636955 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:23.636973 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:23.637019 (+ 46us) server_negotiation.cc:407] Connection header received
0929 18:36:23.637817 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:23.637838 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:23.638237 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:23.638652 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:23.639541 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:23.640654 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:23.641586 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:23.642140 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:23.642336 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:23.645127 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:23.645134 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:23.645151 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:23.645184 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:23.647515 (+ 2331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:23.648083 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:23.648090 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:23.648096 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:23.648274 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:23.648656 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:23.648660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:23.648662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:23.649264 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:23.649490 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:23.649793 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 18:36:23.650150 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":203,"threads_started":1}
I20260929 18:36:23.653436 14121 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3a15cc1414145709f47276e5f3b2211 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3a15cc1414145709f47276e5f3b2211" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 45177 } }
I20260929 18:36:23.653685 14121 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3a15cc1414145709f47276e5f3b2211 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:23.653731 14121 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3a15cc1414145709f47276e5f3b2211 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3a15cc1414145709f47276e5f3b2211, State: Initialized, Role: FOLLOWER
I20260929 18:36:23.654266 14121 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3a15cc1414145709f47276e5f3b2211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3a15cc1414145709f47276e5f3b2211" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 45177 } }
I20260929 18:36:23.654400 14121 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3a15cc1414145709f47276e5f3b2211 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:36:23.654502 14121 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3a15cc1414145709f47276e5f3b2211 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:36:23.654623 14121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3a15cc1414145709f47276e5f3b2211 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:23.655727 14121 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3a15cc1414145709f47276e5f3b2211 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3a15cc1414145709f47276e5f3b2211" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 45177 } }
I20260929 18:36:23.656102 14121 leader_election.cc:302] T 00000000000000000000000000000000 P e3a15cc1414145709f47276e5f3b2211 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3a15cc1414145709f47276e5f3b2211; no voters:
I20260929 18:36:23.656442 14121 leader_election.cc:288] T 00000000000000000000000000000000 P e3a15cc1414145709f47276e5f3b2211 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:36:23.656643 14126 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3a15cc1414145709f47276e5f3b2211 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:23.657047 14126 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3a15cc1414145709f47276e5f3b2211 [term 1 LEADER]: Becoming Leader. State: Replica: e3a15cc1414145709f47276e5f3b2211, State: Running, Role: LEADER
I20260929 18:36:23.657568 14126 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3a15cc1414145709f47276e5f3b2211 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3a15cc1414145709f47276e5f3b2211" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 45177 } }
I20260929 18:36:23.657961 14121 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3a15cc1414145709f47276e5f3b2211 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:36:23.660000 14128 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3a15cc1414145709f47276e5f3b2211 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3a15cc1414145709f47276e5f3b2211. Latest consensus state: current_term: 1 leader_uuid: "e3a15cc1414145709f47276e5f3b2211" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3a15cc1414145709f47276e5f3b2211" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 45177 } } }
I20260929 18:36:23.660163 14128 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3a15cc1414145709f47276e5f3b2211 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:23.660070 14127 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3a15cc1414145709f47276e5f3b2211 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3a15cc1414145709f47276e5f3b2211" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3a15cc1414145709f47276e5f3b2211" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 45177 } } }
I20260929 18:36:23.660271 14127 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3a15cc1414145709f47276e5f3b2211 [sys.catalog]: This master's current role is: LEADER
W20260929 18:36:23.663951 14141 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e3a15cc1414145709f47276e5f3b2211: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:36:23.664055 14141 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:36:23.664129 14140 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:36:23.664911 14140 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:36:23.672178 14140 catalog_manager.cc:1347] Generated new cluster ID: b7426603edf645b4897abade6415673d
I20260929 18:36:23.672272 14140 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:36:23.683495 14140 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:36:23.684926 14140 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:36:23.699517 14140 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3a15cc1414145709f47276e5f3b2211: Generated new TSK 0
I20260929 18:36:23.700598 14140 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.207.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab.
Entry for principal kudu/127.28.207.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab.
I20260929 18:36:23.736667 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.65:0
--local_ip_for_outbound_sockets=127.28.207.65
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:45177
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:34321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:23.883133 14147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:23.883502 14147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:23.883553 14147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:23.888963 14147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:23.889066 14147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:23.889139 14147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.65
I20260929 18:36:23.895794 14147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:34321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:45177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.65
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:23.897226 14147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:23.898466 14147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:23.913015 14152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:23.913031 14155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:23.913031 14153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:23.913434 14147 server_base.cc:1034] running on GCE node
I20260929 18:36:23.914032 14147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:23.914983 14147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:23.916203 14147 hybrid_clock.cc:648] HybridClock initialized: now 1790706983916178 us; error 40 us; skew 500 ppm
Sep 29 18:36:23 dist-test-slave-qcpv krb5kdc[14040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706983, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:23.920483 14147 init.cc:377] Logged in from keytab as kudu/127.28.207.65@KRBTEST.COM (short username kudu)
I20260929 18:36:23.921958 14147 webserver.cc:460] Webserver started at http://127.28.207.65:36047/ using document root <none> and password file <none>
I20260929 18:36:23.922956 14147 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:23.923084 14147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:23.923336 14147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:23.926214 14147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-0/data/instance:
uuid: "4e7854f31f874b56926db6b9a0fe4a00"
format_stamp: "Formatted at 2026-09-29 18:36:23 on dist-test-slave-qcpv"
server_key: "454e4d4531661436ff993021cb6dd210"
server_key_iv: "5bdc8ec0555f9a1ffb7db70830b710b0"
server_key_version: "encryptionkey@0"
I20260929 18:36:23.927074 14147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance:
uuid: "4e7854f31f874b56926db6b9a0fe4a00"
format_stamp: "Formatted at 2026-09-29 18:36:23 on dist-test-slave-qcpv"
server_key: "454e4d4531661436ff993021cb6dd210"
server_key_iv: "5bdc8ec0555f9a1ffb7db70830b710b0"
server_key_version: "encryptionkey@0"
I20260929 18:36:23.932895 14147 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260929 18:36:23.936738 14162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:23.938685 14147 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 18:36:23.938925 14147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-0/wal
uuid: "4e7854f31f874b56926db6b9a0fe4a00"
format_stamp: "Formatted at 2026-09-29 18:36:23 on dist-test-slave-qcpv"
server_key: "454e4d4531661436ff993021cb6dd210"
server_key_iv: "5bdc8ec0555f9a1ffb7db70830b710b0"
server_key_version: "encryptionkey@0"
I20260929 18:36:23.939090 14147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:23.970443 14147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:23.975091 14147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:23.975643 14147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:23.976548 14147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:23.977798 14147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:23.977877 14147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:23.977949 14147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:23.977991 14147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:23.990020 14147 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.65:42707
I20260929 18:36:23.990039 14275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.65:42707 every 8 connection(s)
I20260929 18:36:23.991250 14147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
I20260929 18:36:23.997493 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 14147
I20260929 18:36:23.997611 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance
I20260929 18:36:23.997934 29501 external_mini_cluster.cc:1428] Setting key 6f64676f1b4c3e1cd5b31a0be147f83a
Sep 29 18:36:23 dist-test-slave-qcpv krb5kdc[14040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706983, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.207.66@KRBTEST.COM; defaulting to no policy
I20260929 18:36:24.009917 14124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:23.993518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:36285 (local address 127.28.207.126:45177)
0929 18:36:23.993742 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:23.993747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:23.994737 (+ 990us) server_negotiation.cc:407] Connection header received
0929 18:36:23.996460 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:23.996490 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:23.996552 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:23.996659 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:23.999437 (+ 2778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.000207 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.001626 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.002059 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.002159 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:24.006044 (+ 3885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:24.006047 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:24.006050 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:24.006088 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:24.007642 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:24.008253 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:24.008262 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:24.008263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:24.008346 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:24.008708 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:24.008712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:24.008713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:24.008910 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:24.009015 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:24.009591 (+ 576us) server_negotiation.cc:299] Negotiation successful
0929 18:36:24.009742 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 18:36:24.011288 14278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:23.993826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45177 (local address 127.28.207.65:36285)
0929 18:36:23.994532 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:23.994592 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:23.996171 (+ 1579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:23.997013 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:23.997022 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:23.997634 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:23.999214 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:23.999238 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.000380 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.000384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:24.001474 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:24.001485 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.002260 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.002266 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:24.002560 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:24.003567 (+ 1007us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:24.003594 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:24.005824 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:24.007795 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:24.007802 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:24.007821 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:24.008129 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:24.008442 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:24.008446 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:24.008448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:24.008606 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:24.009026 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:24.009035 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:24.009333 (+ 298us) client_negotiation.cc:769] Sending connection context
0929 18:36:24.009530 (+ 197us) client_negotiation.cc:240] Negotiation successful
0929 18:36:24.009924 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":161,"threads_started":1}
I20260929 18:36:24.013167 14276 heartbeater.cc:347] Connected to a master server at 127.28.207.126:45177
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.66@KRBTEST.COM" created.
I20260929 18:36:24.013643 14276 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:24.014603 14276 heartbeater.cc:511] Master 127.28.207.126:45177 requested a full tablet report, sending...
I20260929 18:36:24.017172 14085 ts_manager.cc:194] Registered new tserver with Master: 4e7854f31f874b56926db6b9a0fe4a00 (127.28.207.65:42707)
I20260929 18:36:24.018555 14085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.65@KRBTEST.COM'} at 127.28.207.65:36285
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab.
Entry for principal kudu/127.28.207.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab.
I20260929 18:36:24.038574 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.66:0
--local_ip_for_outbound_sockets=127.28.207.66
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:45177
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:34321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:24.176131 14281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:24.176463 14281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:24.176554 14281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:24.181171 14281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:24.181293 14281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:24.181391 14281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.66
I20260929 18:36:24.187300 14281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:34321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:45177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.66
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:24.188534 14281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:24.189638 14281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:24.200721 14286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:24.200914 14281 server_base.cc:1034] running on GCE node
W20260929 18:36:24.201094 14289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:24.200982 14287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:24.201906 14281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:24.202777 14281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:24.204013 14281 hybrid_clock.cc:648] HybridClock initialized: now 1790706984203982 us; error 45 us; skew 500 ppm
Sep 29 18:36:24 dist-test-slave-qcpv krb5kdc[14040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706984, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:24.207436 14281 init.cc:377] Logged in from keytab as kudu/127.28.207.66@KRBTEST.COM (short username kudu)
I20260929 18:36:24.208539 14281 webserver.cc:460] Webserver started at http://127.28.207.66:35113/ using document root <none> and password file <none>
I20260929 18:36:24.209259 14281 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:24.209328 14281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:24.209586 14281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:24.212255 14281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-1/data/instance:
uuid: "d7aa24f38ae544b48aa952acc8572b47"
format_stamp: "Formatted at 2026-09-29 18:36:24 on dist-test-slave-qcpv"
server_key: "94007a43c64e761c419fc9f6f05c094e"
server_key_iv: "f10b4f20e88a50d76249e504d21b6bba"
server_key_version: "encryptionkey@0"
I20260929 18:36:24.212934 14281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance:
uuid: "d7aa24f38ae544b48aa952acc8572b47"
format_stamp: "Formatted at 2026-09-29 18:36:24 on dist-test-slave-qcpv"
server_key: "94007a43c64e761c419fc9f6f05c094e"
server_key_iv: "f10b4f20e88a50d76249e504d21b6bba"
server_key_version: "encryptionkey@0"
I20260929 18:36:24.217772 14281 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 18:36:24.221056 14296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:24.222783 14281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:36:24.223013 14281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-1/wal
uuid: "d7aa24f38ae544b48aa952acc8572b47"
format_stamp: "Formatted at 2026-09-29 18:36:24 on dist-test-slave-qcpv"
server_key: "94007a43c64e761c419fc9f6f05c094e"
server_key_iv: "f10b4f20e88a50d76249e504d21b6bba"
server_key_version: "encryptionkey@0"
I20260929 18:36:24.223305 14281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:24.241053 14281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:24.245716 14281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:24.245983 14281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:24.247061 14281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:24.248241 14281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:24.248330 14281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:24.248416 14281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:24.248461 14281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:24.262324 14281 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.66:35149
I20260929 18:36:24.262351 14409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.66:35149 every 8 connection(s)
I20260929 18:36:24.263578 14281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
I20260929 18:36:24.267398 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 14281
I20260929 18:36:24.267539 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance
I20260929 18:36:24.267872 29501 external_mini_cluster.cc:1428] Setting key be2a5069ec645c366bb5e3dcda762364
Sep 29 18:36:24 dist-test-slave-qcpv krb5kdc[14040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706984, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:36:24.281142 14124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:24.265950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:33303 (local address 127.28.207.126:45177)
0929 18:36:24.266187 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:24.266192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:24.267456 (+ 1264us) server_negotiation.cc:407] Connection header received
0929 18:36:24.269275 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:24.269279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:24.269345 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:24.269465 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:24.270984 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.271537 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.273335 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.273642 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.273781 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:24.276820 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:24.276823 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:24.276826 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:24.276855 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:24.278995 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:24.279563 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:24.279566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:24.279568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:24.279654 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:24.279994 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:24.279997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:24.279998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:24.280216 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:24.280321 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:24.280835 (+ 514us) server_negotiation.cc:299] Negotiation successful
0929 18:36:24.280972 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
WARNING: no policy specified for kudu/127.28.207.67@KRBTEST.COM; defaulting to no policy
I20260929 18:36:24.282003 14412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:24.266487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45177 (local address 127.28.207.66:33303)
0929 18:36:24.267249 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:24.267313 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:24.269005 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:24.269749 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:24.269760 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:24.270220 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:24.270771 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:24.270786 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.271677 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.271681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:24.273122 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:24.273134 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.273839 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.273842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:24.273990 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:24.274989 (+ 999us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:24.275006 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:24.276555 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:24.279139 (+ 2584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:24.279146 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:24.279163 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:24.279443 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:24.279745 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:24.279749 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:24.279751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:24.279895 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:24.280319 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:24.280326 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:24.280592 (+ 266us) client_negotiation.cc:769] Sending connection context
0929 18:36:24.280776 (+ 184us) client_negotiation.cc:240] Negotiation successful
0929 18:36:24.281134 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":162,"threads_started":1}
I20260929 18:36:24.283648 14410 heartbeater.cc:347] Connected to a master server at 127.28.207.126:45177
I20260929 18:36:24.283931 14410 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.67@KRBTEST.COM" created.
I20260929 18:36:24.284549 14410 heartbeater.cc:511] Master 127.28.207.126:45177 requested a full tablet report, sending...
I20260929 18:36:24.285816 14085 ts_manager.cc:194] Registered new tserver with Master: d7aa24f38ae544b48aa952acc8572b47 (127.28.207.66:35149)
I20260929 18:36:24.286430 14085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.66@KRBTEST.COM'} at 127.28.207.66:33303
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab.
Entry for principal kudu/127.28.207.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab.
I20260929 18:36:24.302742 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.67:0
--local_ip_for_outbound_sockets=127.28.207.67
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:45177
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:34321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:24.417742 14415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:24.418038 14415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:24.418082 14415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:24.422240 14415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:24.422313 14415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:24.422372 14415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.67
I20260929 18:36:24.427901 14415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:34321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:45177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.67
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:24.429049 14415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:24.430073 14415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:24.440421 14423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:24.440577 14415 server_base.cc:1034] running on GCE node
W20260929 18:36:24.440439 14421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:24.440439 14420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:24.441566 14415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:24.442308 14415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:24.443537 14415 hybrid_clock.cc:648] HybridClock initialized: now 1790706984443481 us; error 76 us; skew 500 ppm
Sep 29 18:36:24 dist-test-slave-qcpv krb5kdc[14040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706984, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:24.447266 14415 init.cc:377] Logged in from keytab as kudu/127.28.207.67@KRBTEST.COM (short username kudu)
I20260929 18:36:24.448433 14415 webserver.cc:460] Webserver started at http://127.28.207.67:34417/ using document root <none> and password file <none>
I20260929 18:36:24.449159 14415 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:24.449262 14415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:24.449534 14415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:24.452113 14415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-2/data/instance:
uuid: "8ae5bf27ce2b462b83bdde49c383fef8"
format_stamp: "Formatted at 2026-09-29 18:36:24 on dist-test-slave-qcpv"
server_key: "b5157392ea837ba58ec5183b19fcc04a"
server_key_iv: "3c85611152fc0f75bf24883809ade30b"
server_key_version: "encryptionkey@0"
I20260929 18:36:24.452750 14415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance:
uuid: "8ae5bf27ce2b462b83bdde49c383fef8"
format_stamp: "Formatted at 2026-09-29 18:36:24 on dist-test-slave-qcpv"
server_key: "b5157392ea837ba58ec5183b19fcc04a"
server_key_iv: "3c85611152fc0f75bf24883809ade30b"
server_key_version: "encryptionkey@0"
I20260929 18:36:24.457643 14415 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 18:36:24.461058 14430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:24.462750 14415 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:36:24.462978 14415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-2/wal
uuid: "8ae5bf27ce2b462b83bdde49c383fef8"
format_stamp: "Formatted at 2026-09-29 18:36:24 on dist-test-slave-qcpv"
server_key: "b5157392ea837ba58ec5183b19fcc04a"
server_key_iv: "3c85611152fc0f75bf24883809ade30b"
server_key_version: "encryptionkey@0"
I20260929 18:36:24.463156 14415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:24.481055 14415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:24.485626 14415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:24.485899 14415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:24.486630 14415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:24.487689 14415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:24.487776 14415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:24.487854 14415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:24.487897 14415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:24.499786 14415 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.67:35085
I20260929 18:36:24.499809 14543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.67:35085 every 8 connection(s)
I20260929 18:36:24.500934 14415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
I20260929 18:36:24.501482 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 14415
I20260929 18:36:24.501643 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance
I20260929 18:36:24.502028 29501 external_mini_cluster.cc:1428] Setting key 9f3f59b8c0a9518fa4ef321133d6ea60
Sep 29 18:36:24 dist-test-slave-qcpv krb5kdc[14040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706984, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:36:24.516991 14124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:24.503282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:36611 (local address 127.28.207.126:45177)
0929 18:36:24.503468 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:24.503473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:24.504765 (+ 1292us) server_negotiation.cc:407] Connection header received
0929 18:36:24.506162 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:24.506165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:24.506221 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:24.506297 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:24.507575 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.508113 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.509247 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.509462 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.509566 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:24.512469 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:24.512474 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:24.512477 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:24.512585 (+ 108us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:24.514264 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:24.514905 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:24.514911 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:24.514913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:24.515025 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:24.515425 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:24.515431 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:24.515433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:24.515692 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:24.515809 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:24.516650 (+ 841us) server_negotiation.cc:299] Negotiation successful
0929 18:36:24.516818 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 18:36:24.518510 14546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:24.503955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45177 (local address 127.28.207.67:36611)
0929 18:36:24.504579 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:24.504643 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:24.505959 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:24.506477 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:24.506490 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:24.506917 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:24.507427 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:24.507440 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.508236 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.508239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:24.509112 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:24.509120 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.509610 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.509614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:24.509780 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:24.510409 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:24.510425 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:24.512193 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:24.514414 (+ 2221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:24.514419 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:24.514433 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:24.514725 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:24.515143 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:24.515147 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:24.515149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:24.515308 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:24.515874 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:24.515885 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:24.516325 (+ 440us) client_negotiation.cc:769] Sending connection context
0929 18:36:24.516581 (+ 256us) client_negotiation.cc:240] Negotiation successful
0929 18:36:24.517069 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":166,"threads_started":1}
I20260929 18:36:24.520628 14544 heartbeater.cc:347] Connected to a master server at 127.28.207.126:45177
I20260929 18:36:24.520939 14544 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:24.521623 14544 heartbeater.cc:511] Master 127.28.207.126:45177 requested a full tablet report, sending...
I20260929 18:36:24.522732 14085 ts_manager.cc:194] Registered new tserver with Master: 8ae5bf27ce2b462b83bdde49c383fef8 (127.28.207.67:35085)
I20260929 18:36:24.523373 14085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.67@KRBTEST.COM'} at 127.28.207.67:36611
I20260929 18:36:24.529644 29501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:36:24.541672 14124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:24.532605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47894 (local address 127.28.207.126:45177)
0929 18:36:24.532840 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:24.532846 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:24.532976 (+ 130us) server_negotiation.cc:407] Connection header received
0929 18:36:24.533153 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:24.533157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:24.533224 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:24.533330 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:24.534536 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.535322 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.536264 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.536508 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.536619 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:24.538236 (+ 1617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:24.538240 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:24.538242 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:24.538266 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:24.539822 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:24.540309 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:24.540314 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:24.540316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:24.540402 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:24.540677 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:24.540687 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:24.540689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:24.540953 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:24.541106 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:24.541352 (+ 246us) server_negotiation.cc:299] Negotiation successful
0929 18:36:24.541480 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260929 18:36:24.545830 14085 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47894:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:36:24.550019 14085 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:36:24.567601 14557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:24.561324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:35149 (local address 127.0.0.1:60542)
0929 18:36:24.562174 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:24.562218 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:24.562398 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:24.563182 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:24.563189 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:24.563213 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:24.563351 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:24.563356 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.564736 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.564740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:24.565740 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:24.565749 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.566845 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.566849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:24.567020 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:24.567072 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 18:36:24.567203 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 18:36:24.567386 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"mutex_wait_us":3,"thread_start_us":129,"threads_started":1}
I20260929 18:36:24.567662 14556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:24.561434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60542 (local address 127.28.207.66:35149)
0929 18:36:24.561905 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:24.561915 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:24.562325 (+ 410us) server_negotiation.cc:407] Connection header received
0929 18:36:24.562557 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:24.562563 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:24.562792 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:24.562970 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:24.563476 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.564573 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.565863 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.566606 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.566731 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:24.566903 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:24.567319 (+ 416us) server_negotiation.cc:299] Negotiation successful
0929 18:36:24.567434 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":139,"threads_started":1}
I20260929 18:36:24.569639 14555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:24.561479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:42707 (local address 127.0.0.1:53136)
0929 18:36:24.562979 (+ 1500us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:24.562998 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:24.563116 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:24.564492 (+ 1376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:24.564495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:24.564513 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:24.564711 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:24.564723 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.566211 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.566215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:24.567486 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:24.567498 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.569162 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.569165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:24.569264 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:24.569290 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 18:36:24.569368 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 18:36:24.569455 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1415,"mutex_wait_us":88,"thread_start_us":451,"threads_started":1}
I20260929 18:36:24.569762 14558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:24.562627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53136 (local address 127.28.207.65:42707)
0929 18:36:24.563879 (+ 1252us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:24.563887 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:24.563914 (+ 27us) server_negotiation.cc:407] Connection header received
0929 18:36:24.564002 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:24.564010 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:24.564322 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:24.564674 (+ 352us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:24.564832 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.566060 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.568100 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.569018 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.569150 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:24.569289 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:24.569476 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 18:36:24.569602 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1113,"thread_start_us":143,"threads_started":1}
I20260929 18:36:24.570335 14559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:24.563025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:35085 (local address 127.0.0.1:42040)
0929 18:36:24.563471 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:24.563492 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:24.563600 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:24.564514 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:24.564517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:24.564536 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:24.564839 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:24.564845 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.566298 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.566301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:24.568140 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:24.568151 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.569376 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.569379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:24.569469 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:24.569497 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 18:36:24.569566 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 18:36:24.569643 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":7,"thread_start_us":114,"threads_started":1}
I20260929 18:36:24.570977 14560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:24.563367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42040 (local address 127.28.207.67:35085)
0929 18:36:24.563948 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:24.563962 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:24.563985 (+ 23us) server_negotiation.cc:407] Connection header received
0929 18:36:24.564055 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:24.564059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:24.564236 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:24.564374 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:24.564968 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.566121 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.568297 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.569107 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.569215 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:24.569373 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:24.570660 (+ 1287us) server_negotiation.cc:299] Negotiation successful
0929 18:36:24.570793 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":154,"threads_started":1}
I20260929 18:36:24.572515 14344 tablet_service.cc:1467] Processing CreateTablet for tablet c57df591db63490eae3872b9cfcc6302 (DEFAULT_TABLE table=test-table [id=ff107a12ce644073b028310bc7b4b034]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:24.573269 14210 tablet_service.cc:1467] Processing CreateTablet for tablet c57df591db63490eae3872b9cfcc6302 (DEFAULT_TABLE table=test-table [id=ff107a12ce644073b028310bc7b4b034]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:24.574282 14210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c57df591db63490eae3872b9cfcc6302. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:24.574143 14344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c57df591db63490eae3872b9cfcc6302. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:24.573899 14478 tablet_service.cc:1467] Processing CreateTablet for tablet c57df591db63490eae3872b9cfcc6302 (DEFAULT_TABLE table=test-table [id=ff107a12ce644073b028310bc7b4b034]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:24.575313 14478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c57df591db63490eae3872b9cfcc6302. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:24.580926 14561 tablet_bootstrap.cc:492] T c57df591db63490eae3872b9cfcc6302 P 4e7854f31f874b56926db6b9a0fe4a00: Bootstrap starting.
I20260929 18:36:24.583670 14561 tablet_bootstrap.cc:654] T c57df591db63490eae3872b9cfcc6302 P 4e7854f31f874b56926db6b9a0fe4a00: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:24.583781 14562 tablet_bootstrap.cc:492] T c57df591db63490eae3872b9cfcc6302 P 8ae5bf27ce2b462b83bdde49c383fef8: Bootstrap starting.
I20260929 18:36:24.584607 14561 log.cc:824] T c57df591db63490eae3872b9cfcc6302 P 4e7854f31f874b56926db6b9a0fe4a00: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:24.585451 14563 tablet_bootstrap.cc:492] T c57df591db63490eae3872b9cfcc6302 P d7aa24f38ae544b48aa952acc8572b47: Bootstrap starting.
I20260929 18:36:24.586050 14562 tablet_bootstrap.cc:654] T c57df591db63490eae3872b9cfcc6302 P 8ae5bf27ce2b462b83bdde49c383fef8: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:24.587077 14561 tablet_bootstrap.cc:492] T c57df591db63490eae3872b9cfcc6302 P 4e7854f31f874b56926db6b9a0fe4a00: No bootstrap required, opened a new log
I20260929 18:36:24.587328 14561 ts_tablet_manager.cc:1397] T c57df591db63490eae3872b9cfcc6302 P 4e7854f31f874b56926db6b9a0fe4a00: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 18:36:24.587424 14562 log.cc:824] T c57df591db63490eae3872b9cfcc6302 P 8ae5bf27ce2b462b83bdde49c383fef8: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:24.587648 14563 tablet_bootstrap.cc:654] T c57df591db63490eae3872b9cfcc6302 P d7aa24f38ae544b48aa952acc8572b47: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:24.588763 14563 log.cc:824] T c57df591db63490eae3872b9cfcc6302 P d7aa24f38ae544b48aa952acc8572b47: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:24.589778 14562 tablet_bootstrap.cc:492] T c57df591db63490eae3872b9cfcc6302 P 8ae5bf27ce2b462b83bdde49c383fef8: No bootstrap required, opened a new log
I20260929 18:36:24.589983 14562 ts_tablet_manager.cc:1397] T c57df591db63490eae3872b9cfcc6302 P 8ae5bf27ce2b462b83bdde49c383fef8: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20260929 18:36:24.590962 14563 tablet_bootstrap.cc:492] T c57df591db63490eae3872b9cfcc6302 P d7aa24f38ae544b48aa952acc8572b47: No bootstrap required, opened a new log
I20260929 18:36:24.591140 14563 ts_tablet_manager.cc:1397] T c57df591db63490eae3872b9cfcc6302 P d7aa24f38ae544b48aa952acc8572b47: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 18:36:24.592916 14561 raft_consensus.cc:348] T c57df591db63490eae3872b9cfcc6302 P 4e7854f31f874b56926db6b9a0fe4a00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7aa24f38ae544b48aa952acc8572b47" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 35149 } } peers { permanent_uuid: "4e7854f31f874b56926db6b9a0fe4a00" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 42707 } } peers { permanent_uuid: "8ae5bf27ce2b462b83bdde49c383fef8" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 35085 } }
I20260929 18:36:24.593245 14561 raft_consensus.cc:374] T c57df591db63490eae3872b9cfcc6302 P 4e7854f31f874b56926db6b9a0fe4a00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:24.593304 14561 raft_consensus.cc:729] T c57df591db63490eae3872b9cfcc6302 P 4e7854f31f874b56926db6b9a0fe4a00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e7854f31f874b56926db6b9a0fe4a00, State: Initialized, Role: FOLLOWER
I20260929 18:36:24.593981 14561 consensus_queue.cc:260] T c57df591db63490eae3872b9cfcc6302 P 4e7854f31f874b56926db6b9a0fe4a00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7aa24f38ae544b48aa952acc8572b47" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 35149 } } peers { permanent_uuid: "4e7854f31f874b56926db6b9a0fe4a00" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 42707 } } peers { permanent_uuid: "8ae5bf27ce2b462b83bdde49c383fef8" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 35085 } }
I20260929 18:36:24.593971 14562 raft_consensus.cc:348] T c57df591db63490eae3872b9cfcc6302 P 8ae5bf27ce2b462b83bdde49c383fef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7aa24f38ae544b48aa952acc8572b47" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 35149 } } peers { permanent_uuid: "4e7854f31f874b56926db6b9a0fe4a00" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 42707 } } peers { permanent_uuid: "8ae5bf27ce2b462b83bdde49c383fef8" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 35085 } }
I20260929 18:36:24.594275 14562 raft_consensus.cc:374] T c57df591db63490eae3872b9cfcc6302 P 8ae5bf27ce2b462b83bdde49c383fef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:24.594342 14562 raft_consensus.cc:729] T c57df591db63490eae3872b9cfcc6302 P 8ae5bf27ce2b462b83bdde49c383fef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae5bf27ce2b462b83bdde49c383fef8, State: Initialized, Role: FOLLOWER
I20260929 18:36:24.594897 14562 consensus_queue.cc:260] T c57df591db63490eae3872b9cfcc6302 P 8ae5bf27ce2b462b83bdde49c383fef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7aa24f38ae544b48aa952acc8572b47" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 35149 } } peers { permanent_uuid: "4e7854f31f874b56926db6b9a0fe4a00" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 42707 } } peers { permanent_uuid: "8ae5bf27ce2b462b83bdde49c383fef8" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 35085 } }
I20260929 18:36:24.594897 14563 raft_consensus.cc:348] T c57df591db63490eae3872b9cfcc6302 P d7aa24f38ae544b48aa952acc8572b47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7aa24f38ae544b48aa952acc8572b47" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 35149 } } peers { permanent_uuid: "4e7854f31f874b56926db6b9a0fe4a00" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 42707 } } peers { permanent_uuid: "8ae5bf27ce2b462b83bdde49c383fef8" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 35085 } }
I20260929 18:36:24.595142 14563 raft_consensus.cc:374] T c57df591db63490eae3872b9cfcc6302 P d7aa24f38ae544b48aa952acc8572b47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:24.595227 14563 raft_consensus.cc:729] T c57df591db63490eae3872b9cfcc6302 P d7aa24f38ae544b48aa952acc8572b47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7aa24f38ae544b48aa952acc8572b47, State: Initialized, Role: FOLLOWER
I20260929 18:36:24.595237 14561 ts_tablet_manager.cc:1428] T c57df591db63490eae3872b9cfcc6302 P 4e7854f31f874b56926db6b9a0fe4a00: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260929 18:36:24.595609 14276 heartbeater.cc:503] Master 127.28.207.126:45177 was elected leader, sending a full tablet report...
I20260929 18:36:24.595942 14562 ts_tablet_manager.cc:1428] T c57df591db63490eae3872b9cfcc6302 P 8ae5bf27ce2b462b83bdde49c383fef8: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260929 18:36:24.596376 14544 heartbeater.cc:503] Master 127.28.207.126:45177 was elected leader, sending a full tablet report...
I20260929 18:36:24.596836 14563 consensus_queue.cc:260] T c57df591db63490eae3872b9cfcc6302 P d7aa24f38ae544b48aa952acc8572b47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7aa24f38ae544b48aa952acc8572b47" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 35149 } } peers { permanent_uuid: "4e7854f31f874b56926db6b9a0fe4a00" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 42707 } } peers { permanent_uuid: "8ae5bf27ce2b462b83bdde49c383fef8" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 35085 } }
I20260929 18:36:24.597494 14410 heartbeater.cc:503] Master 127.28.207.126:45177 was elected leader, sending a full tablet report...
I20260929 18:36:24.597970 14563 ts_tablet_manager.cc:1428] T c57df591db63490eae3872b9cfcc6302 P d7aa24f38ae544b48aa952acc8572b47: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260929 18:36:24.604305 14568 raft_consensus.cc:482] T c57df591db63490eae3872b9cfcc6302 P 8ae5bf27ce2b462b83bdde49c383fef8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:36:24.604561 14568 raft_consensus.cc:504] T c57df591db63490eae3872b9cfcc6302 P 8ae5bf27ce2b462b83bdde49c383fef8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7aa24f38ae544b48aa952acc8572b47" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 35149 } } peers { permanent_uuid: "4e7854f31f874b56926db6b9a0fe4a00" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 42707 } } peers { permanent_uuid: "8ae5bf27ce2b462b83bdde49c383fef8" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 35085 } }
I20260929 18:36:24.606066 14568 leader_election.cc:288] T c57df591db63490eae3872b9cfcc6302 P 8ae5bf27ce2b462b83bdde49c383fef8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7aa24f38ae544b48aa952acc8572b47 (127.28.207.66:35149), 4e7854f31f874b56926db6b9a0fe4a00 (127.28.207.65:42707)
I20260929 18:36:24.611088 14570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:24.606369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:35149 (local address 127.28.207.67:42639)
0929 18:36:24.607083 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:24.607097 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:24.607245 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:24.607528 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:24.607531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:24.607552 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:24.607730 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:24.607736 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.608529 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.608533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:24.609749 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:24.609760 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.610553 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.610557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:24.610662 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:24.610690 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 18:36:24.610806 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 18:36:24.610916 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"mutex_wait_us":395,"thread_start_us":194,"threads_started":1}
I20260929 18:36:24.611063 14556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:24.606679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:42639 (local address 127.28.207.66:35149)
0929 18:36:24.606814 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:24.606822 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:24.607184 (+ 362us) server_negotiation.cc:407] Connection header received
0929 18:36:24.607372 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:24.607375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:24.607427 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:24.607515 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:24.607850 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.608398 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.609889 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.610425 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.610564 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:24.610627 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:24.610823 (+ 196us) server_negotiation.cc:299] Negotiation successful
0929 18:36:24.610922 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 18:36:24.611433 14546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:24.606370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:42707 (local address 127.28.207.67:35297)
0929 18:36:24.606666 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:24.606711 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:24.606986 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:24.607388 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:24.607391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:24.607422 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:24.607580 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:24.607586 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.608587 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.608590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:24.609777 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:24.609788 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.610908 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.610912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:24.611033 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:24.611060 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 18:36:24.611163 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 18:36:24.611279 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":44}
I20260929 18:36:24.611742 14558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:24.606627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:35297 (local address 127.28.207.65:42707)
0929 18:36:24.606951 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:24.606957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:24.606975 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:36:24.607221 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:24.607229 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:24.607283 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:24.607376 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:24.607688 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.608440 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.609925 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.610536 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.610948 (+ 412us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:24.611015 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:24.611539 (+ 524us) server_negotiation.cc:299] Negotiation successful
0929 18:36:24.611630 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 18:36:24.611832 14364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c57df591db63490eae3872b9cfcc6302" candidate_uuid: "8ae5bf27ce2b462b83bdde49c383fef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7aa24f38ae544b48aa952acc8572b47" is_pre_election: true
I20260929 18:36:24.612215 14364 raft_consensus.cc:2454] T c57df591db63490eae3872b9cfcc6302 P d7aa24f38ae544b48aa952acc8572b47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ae5bf27ce2b462b83bdde49c383fef8 in term 0.
I20260929 18:36:24.612943 14431 leader_election.cc:302] T c57df591db63490eae3872b9cfcc6302 P 8ae5bf27ce2b462b83bdde49c383fef8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ae5bf27ce2b462b83bdde49c383fef8, d7aa24f38ae544b48aa952acc8572b47; no voters:
I20260929 18:36:24.613015 14230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c57df591db63490eae3872b9cfcc6302" candidate_uuid: "8ae5bf27ce2b462b83bdde49c383fef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e7854f31f874b56926db6b9a0fe4a00" is_pre_election: true
I20260929 18:36:24.613301 14230 raft_consensus.cc:2454] T c57df591db63490eae3872b9cfcc6302 P 4e7854f31f874b56926db6b9a0fe4a00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ae5bf27ce2b462b83bdde49c383fef8 in term 0.
I20260929 18:36:24.613307 14568 raft_consensus.cc:2781] T c57df591db63490eae3872b9cfcc6302 P 8ae5bf27ce2b462b83bdde49c383fef8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:36:24.613401 14568 raft_consensus.cc:482] T c57df591db63490eae3872b9cfcc6302 P 8ae5bf27ce2b462b83bdde49c383fef8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:36:24.613464 14568 raft_consensus.cc:3037] T c57df591db63490eae3872b9cfcc6302 P 8ae5bf27ce2b462b83bdde49c383fef8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:24.614714 14568 raft_consensus.cc:504] T c57df591db63490eae3872b9cfcc6302 P 8ae5bf27ce2b462b83bdde49c383fef8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7aa24f38ae544b48aa952acc8572b47" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 35149 } } peers { permanent_uuid: "4e7854f31f874b56926db6b9a0fe4a00" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 42707 } } peers { permanent_uuid: "8ae5bf27ce2b462b83bdde49c383fef8" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 35085 } }
I20260929 18:36:24.615177 14568 leader_election.cc:288] T c57df591db63490eae3872b9cfcc6302 P 8ae5bf27ce2b462b83bdde49c383fef8 [CANDIDATE]: Term 1 election: Requested vote from peers d7aa24f38ae544b48aa952acc8572b47 (127.28.207.66:35149), 4e7854f31f874b56926db6b9a0fe4a00 (127.28.207.65:42707)
I20260929 18:36:24.615684 14230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c57df591db63490eae3872b9cfcc6302" candidate_uuid: "8ae5bf27ce2b462b83bdde49c383fef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e7854f31f874b56926db6b9a0fe4a00"
I20260929 18:36:24.615806 14230 raft_consensus.cc:3037] T c57df591db63490eae3872b9cfcc6302 P 4e7854f31f874b56926db6b9a0fe4a00 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:24.617010 14230 raft_consensus.cc:2454] T c57df591db63490eae3872b9cfcc6302 P 4e7854f31f874b56926db6b9a0fe4a00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ae5bf27ce2b462b83bdde49c383fef8 in term 1.
I20260929 18:36:24.617331 14364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c57df591db63490eae3872b9cfcc6302" candidate_uuid: "8ae5bf27ce2b462b83bdde49c383fef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7aa24f38ae544b48aa952acc8572b47"
I20260929 18:36:24.617498 14434 leader_election.cc:302] T c57df591db63490eae3872b9cfcc6302 P 8ae5bf27ce2b462b83bdde49c383fef8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e7854f31f874b56926db6b9a0fe4a00, 8ae5bf27ce2b462b83bdde49c383fef8; no voters:
I20260929 18:36:24.617506 14364 raft_consensus.cc:3037] T c57df591db63490eae3872b9cfcc6302 P d7aa24f38ae544b48aa952acc8572b47 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:24.617849 14568 raft_consensus.cc:2781] T c57df591db63490eae3872b9cfcc6302 P 8ae5bf27ce2b462b83bdde49c383fef8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:24.618180 14568 raft_consensus.cc:686] T c57df591db63490eae3872b9cfcc6302 P 8ae5bf27ce2b462b83bdde49c383fef8 [term 1 LEADER]: Becoming Leader. State: Replica: 8ae5bf27ce2b462b83bdde49c383fef8, State: Running, Role: LEADER
I20260929 18:36:24.618832 14364 raft_consensus.cc:2454] T c57df591db63490eae3872b9cfcc6302 P d7aa24f38ae544b48aa952acc8572b47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ae5bf27ce2b462b83bdde49c383fef8 in term 1.
I20260929 18:36:24.618767 14568 consensus_queue.cc:237] T c57df591db63490eae3872b9cfcc6302 P 8ae5bf27ce2b462b83bdde49c383fef8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7aa24f38ae544b48aa952acc8572b47" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 35149 } } peers { permanent_uuid: "4e7854f31f874b56926db6b9a0fe4a00" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 42707 } } peers { permanent_uuid: "8ae5bf27ce2b462b83bdde49c383fef8" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 35085 } }
I20260929 18:36:24.623440 14085 catalog_manager.cc:5495] T c57df591db63490eae3872b9cfcc6302 P 8ae5bf27ce2b462b83bdde49c383fef8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ae5bf27ce2b462b83bdde49c383fef8 (127.28.207.67). New cstate: current_term: 1 leader_uuid: "8ae5bf27ce2b462b83bdde49c383fef8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7aa24f38ae544b48aa952acc8572b47" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 35149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e7854f31f874b56926db6b9a0fe4a00" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 42707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ae5bf27ce2b462b83bdde49c383fef8" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 35085 } health_report { overall_health: HEALTHY } } }
I20260929 18:36:24.649163 14560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:24.645469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42052 (local address 127.28.207.67:35085)
0929 18:36:24.645628 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:24.645633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:24.645660 (+ 27us) server_negotiation.cc:407] Connection header received
0929 18:36:24.645749 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:24.645752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:24.645807 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:24.645913 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:36:24.646183 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.646725 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.647647 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.647858 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.647960 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:24.648279 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:36:24.648741 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:36:24.648866 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:24.648924 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 18:36:24.648982 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 18:36:24.656227 14230 raft_consensus.cc:1261] T c57df591db63490eae3872b9cfcc6302 P 4e7854f31f874b56926db6b9a0fe4a00 [term 1 FOLLOWER]: Refusing update from remote peer 8ae5bf27ce2b462b83bdde49c383fef8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:36:24.656606 14364 raft_consensus.cc:1261] T c57df591db63490eae3872b9cfcc6302 P d7aa24f38ae544b48aa952acc8572b47 [term 1 FOLLOWER]: Refusing update from remote peer 8ae5bf27ce2b462b83bdde49c383fef8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:36:24.657033 14571 consensus_queue.cc:1035] T c57df591db63490eae3872b9cfcc6302 P 8ae5bf27ce2b462b83bdde49c383fef8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e7854f31f874b56926db6b9a0fe4a00" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 42707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:36:24.657508 14568 consensus_queue.cc:1035] T c57df591db63490eae3872b9cfcc6302 P 8ae5bf27ce2b462b83bdde49c383fef8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7aa24f38ae544b48aa952acc8572b47" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 35149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:36:24.664711 14574 mvcc.cc:205] Tried to move back new op lower bound from 7334735809142845440 to 7334735809002762240. Current Snapshot: MvccSnapshot[applied={T|T < 7334735809142845440}]
I20260929 18:36:24.666356 14575 mvcc.cc:205] Tried to move back new op lower bound from 7334735809142845440 to 7334735809002762240. Current Snapshot: MvccSnapshot[applied={T|T < 7334735809142845440}]
I20260929 18:36:24.673883 14084 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:36:24.674130 14084 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:36:24.674552 14576 mvcc.cc:205] Tried to move back new op lower bound from 7334735809142845440 to 7334735809002762240. Current Snapshot: MvccSnapshot[applied={T|T < 7334735809142845440}]
I20260929 18:36:24.677553 14084 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e3a15cc1414145709f47276e5f3b2211: Sending DeleteTablet for 3 replicas of tablet c57df591db63490eae3872b9cfcc6302
I20260929 18:36:24.678874 14344 tablet_service.cc:1514] Processing DeleteTablet for tablet c57df591db63490eae3872b9cfcc6302 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:24 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:60542
I20260929 18:36:24.678849 14210 tablet_service.cc:1514] Processing DeleteTablet for tablet c57df591db63490eae3872b9cfcc6302 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:24 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:53136
I20260929 18:36:24.679625 14584 tablet_replica.cc:330] T c57df591db63490eae3872b9cfcc6302 P d7aa24f38ae544b48aa952acc8572b47: stopping tablet replica
I20260929 18:36:24.679643 14478 tablet_service.cc:1514] Processing DeleteTablet for tablet c57df591db63490eae3872b9cfcc6302 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:24 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:42040
I20260929 18:36:24.680190 14584 raft_consensus.cc:2229] T c57df591db63490eae3872b9cfcc6302 P d7aa24f38ae544b48aa952acc8572b47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:36:24.680547 14586 tablet_replica.cc:330] T c57df591db63490eae3872b9cfcc6302 P 8ae5bf27ce2b462b83bdde49c383fef8: stopping tablet replica
I20260929 18:36:24.680652 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 14147
I20260929 18:36:24.680744 14584 raft_consensus.cc:2258] T c57df591db63490eae3872b9cfcc6302 P d7aa24f38ae544b48aa952acc8572b47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:36:24.683557 14586 raft_consensus.cc:2229] T c57df591db63490eae3872b9cfcc6302 P 8ae5bf27ce2b462b83bdde49c383fef8 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:36:24.684065 14586 raft_consensus.cc:2258] T c57df591db63490eae3872b9cfcc6302 P 8ae5bf27ce2b462b83bdde49c383fef8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:36:24.685182 14584 ts_tablet_manager.cc:1905] T c57df591db63490eae3872b9cfcc6302 P d7aa24f38ae544b48aa952acc8572b47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:36:24.689508 14584 ts_tablet_manager.cc:1918] T c57df591db63490eae3872b9cfcc6302 P d7aa24f38ae544b48aa952acc8572b47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:36:24.689648 14584 log.cc:1196] T c57df591db63490eae3872b9cfcc6302 P d7aa24f38ae544b48aa952acc8572b47: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-1/wal/wals/c57df591db63490eae3872b9cfcc6302
I20260929 18:36:24.690054 14584 ts_tablet_manager.cc:1939] T c57df591db63490eae3872b9cfcc6302 P d7aa24f38ae544b48aa952acc8572b47: Deleting consensus metadata
I20260929 18:36:24.691720 14586 ts_tablet_manager.cc:1905] T c57df591db63490eae3872b9cfcc6302 P 8ae5bf27ce2b462b83bdde49c383fef8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:36:24.692286 14069 catalog_manager.cc:4841] TS d7aa24f38ae544b48aa952acc8572b47 (127.28.207.66:35149): tablet c57df591db63490eae3872b9cfcc6302 (table test-table [id=ff107a12ce644073b028310bc7b4b034]) successfully deleted
W20260929 18:36:24.693030 14072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.207.65:42707 (error 108)
I20260929 18:36:24.693784 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 14281
I20260929 18:36:24.699527 14586 ts_tablet_manager.cc:1918] T c57df591db63490eae3872b9cfcc6302 P 8ae5bf27ce2b462b83bdde49c383fef8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:36:24.700098 14559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:24.699662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:42707 (local address 127.0.0.1:53144)
0929 18:36:24.699868 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:24.699987 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.65:42707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 18:36:24.700580 14072 catalog_manager.cc:4584] TS 4e7854f31f874b56926db6b9a0fe4a00 (127.28.207.65:42707): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c57df591db63490eae3872b9cfcc6302: Network error: Client connection negotiation failed: client connection to 127.28.207.65:42707: connect: Connection refused (error 111)
I20260929 18:36:24.701211 14586 log.cc:1196] T c57df591db63490eae3872b9cfcc6302 P 8ae5bf27ce2b462b83bdde49c383fef8: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790706718260899-29501-0/minicluster-data/ts-2/wal/wals/c57df591db63490eae3872b9cfcc6302
I20260929 18:36:24.701424 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 14415
I20260929 18:36:24.707958 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 14053
2026-09-29T18:36:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2507 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T18:36:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:36:24Z Disabled control of system clock
I20260929 18:36:24.744498 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:45729
--webserver_interface=127.28.207.126
--webserver_port=0
--builtin_ntp_servers=127.28.207.84:39345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:45729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:36:24.861446 14591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:24.861770 14591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:24.861809 14591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:24.866142 14591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:36:24.866230 14591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:24.866246 14591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:24.866263 14591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:36:24.866279 14591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:36:24.871510 14591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:39345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:45729
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:45729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:24.872642 14591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:24.874038 14591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:24.884199 14597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:24.884428 14596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:24.884199 14599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:24.885108 14591 server_base.cc:1034] running on GCE node
I20260929 18:36:24.885871 14591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:24.887229 14591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:24.888466 14591 hybrid_clock.cc:648] HybridClock initialized: now 1790706984888423 us; error 60 us; skew 500 ppm
I20260929 18:36:24.890949 14591 webserver.cc:460] Webserver started at http://127.28.207.126:36277/ using document root <none> and password file <none>
I20260929 18:36:24.891647 14591 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:24.891739 14591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:24.891978 14591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:24.894260 14591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "0283f35103594cecbd48c1ae80fbaeed"
format_stamp: "Formatted at 2026-09-29 18:36:24 on dist-test-slave-qcpv"
server_key: "b9ccaa65476a448afdcd87c043b4943a"
server_key_iv: "97892efe78e15a59639fa22bf413f76e"
server_key_version: "encryptionkey@0"
I20260929 18:36:24.894979 14591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "0283f35103594cecbd48c1ae80fbaeed"
format_stamp: "Formatted at 2026-09-29 18:36:24 on dist-test-slave-qcpv"
server_key: "b9ccaa65476a448afdcd87c043b4943a"
server_key_iv: "97892efe78e15a59639fa22bf413f76e"
server_key_version: "encryptionkey@0"
I20260929 18:36:24.899461 14591 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:36:24.902508 14605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:24.904421 14591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:36:24.904639 14591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "0283f35103594cecbd48c1ae80fbaeed"
format_stamp: "Formatted at 2026-09-29 18:36:24 on dist-test-slave-qcpv"
server_key: "b9ccaa65476a448afdcd87c043b4943a"
server_key_iv: "97892efe78e15a59639fa22bf413f76e"
server_key_version: "encryptionkey@0"
I20260929 18:36:24.904776 14591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:24.922127 14591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:24.923185 14591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:24.923425 14591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:24.932449 14591 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:45729
I20260929 18:36:24.932489 14657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:45729 every 8 connection(s)
I20260929 18:36:24.933988 14591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:36:24.938014 14658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:24.942363 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 14591
I20260929 18:36:24.942528 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:36:24.943032 29501 external_mini_cluster.cc:1428] Setting key 93e6804f6d406ea0d7e7adea699ebe10
I20260929 18:36:24.944504 14658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0283f35103594cecbd48c1ae80fbaeed: Bootstrap starting.
I20260929 18:36:24.947468 14658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0283f35103594cecbd48c1ae80fbaeed: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:24.948402 14658 log.cc:824] T 00000000000000000000000000000000 P 0283f35103594cecbd48c1ae80fbaeed: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:24.951502 14658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0283f35103594cecbd48c1ae80fbaeed: No bootstrap required, opened a new log
I20260929 18:36:24.955935 14661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:24.945156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36642 (local address 127.28.207.126:45729)
0929 18:36:24.946037 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:24.946056 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:24.946107 (+ 51us) server_negotiation.cc:407] Connection header received
0929 18:36:24.947101 (+ 994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:24.947127 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:24.947695 (+ 568us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:24.948127 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:24.948970 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.950120 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.951543 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:24.952226 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:24.952384 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:24.953641 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:24.953646 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:24.953663 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:24.954155 (+ 492us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:24.954231 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:24.954256 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:24.954364 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:24.954719 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 18:36:24.955077 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":153,"threads_started":1}
I20260929 18:36:24.957273 14658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0283f35103594cecbd48c1ae80fbaeed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0283f35103594cecbd48c1ae80fbaeed" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 45729 } }
I20260929 18:36:24.957731 14658 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0283f35103594cecbd48c1ae80fbaeed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:24.957804 14658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0283f35103594cecbd48c1ae80fbaeed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0283f35103594cecbd48c1ae80fbaeed, State: Initialized, Role: FOLLOWER
I20260929 18:36:24.958423 14658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0283f35103594cecbd48c1ae80fbaeed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0283f35103594cecbd48c1ae80fbaeed" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 45729 } }
I20260929 18:36:24.958573 14658 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0283f35103594cecbd48c1ae80fbaeed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:36:24.958621 14658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0283f35103594cecbd48c1ae80fbaeed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:36:24.958701 14658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0283f35103594cecbd48c1ae80fbaeed [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:24.960196 14658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0283f35103594cecbd48c1ae80fbaeed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0283f35103594cecbd48c1ae80fbaeed" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 45729 } }
I20260929 18:36:24.960611 14658 leader_election.cc:302] T 00000000000000000000000000000000 P 0283f35103594cecbd48c1ae80fbaeed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0283f35103594cecbd48c1ae80fbaeed; no voters:
I20260929 18:36:24.961464 14658 leader_election.cc:288] T 00000000000000000000000000000000 P 0283f35103594cecbd48c1ae80fbaeed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:36:24.961781 14663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0283f35103594cecbd48c1ae80fbaeed [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:24.962383 14663 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0283f35103594cecbd48c1ae80fbaeed [term 1 LEADER]: Becoming Leader. State: Replica: 0283f35103594cecbd48c1ae80fbaeed, State: Running, Role: LEADER
I20260929 18:36:24.963080 14663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0283f35103594cecbd48c1ae80fbaeed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0283f35103594cecbd48c1ae80fbaeed" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 45729 } }
I20260929 18:36:24.963701 14658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0283f35103594cecbd48c1ae80fbaeed [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:36:24.964831 14664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0283f35103594cecbd48c1ae80fbaeed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0283f35103594cecbd48c1ae80fbaeed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0283f35103594cecbd48c1ae80fbaeed" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 45729 } } }
I20260929 18:36:24.964979 14664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0283f35103594cecbd48c1ae80fbaeed [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:24.965361 14665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0283f35103594cecbd48c1ae80fbaeed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0283f35103594cecbd48c1ae80fbaeed. Latest consensus state: current_term: 1 leader_uuid: "0283f35103594cecbd48c1ae80fbaeed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0283f35103594cecbd48c1ae80fbaeed" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 45729 } } }
I20260929 18:36:24.965474 14665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0283f35103594cecbd48c1ae80fbaeed [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:24.965767 14668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:36:24.969379 14668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:36:24.977280 14668 catalog_manager.cc:1347] Generated new cluster ID: addcab93e27c4ce3b7fe7bda81503cad
I20260929 18:36:24.977401 14668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:36:24.986419 14668 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:36:24.987623 14668 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:36:24.995479 14668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0283f35103594cecbd48c1ae80fbaeed: Generated new TSK 0
I20260929 18:36:24.996436 14668 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:36:25.000749 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.65:0
--local_ip_for_outbound_sockets=127.28.207.65
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:45729
--builtin_ntp_servers=127.28.207.84:39345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:36:25.162206 14682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:25.162580 14682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:25.162624 14682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:25.168605 14682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:25.168702 14682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:25.168774 14682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.65
I20260929 18:36:25.174153 14682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:39345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:45729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.65
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:25.175402 14682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:25.176905 14682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:25.187709 14687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:25.187714 14688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:25.187789 14690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:25.188261 14682 server_base.cc:1034] running on GCE node
I20260929 18:36:25.188764 14682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:25.189551 14682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:25.190759 14682 hybrid_clock.cc:648] HybridClock initialized: now 1790706985190712 us; error 55 us; skew 500 ppm
I20260929 18:36:25.193488 14682 webserver.cc:460] Webserver started at http://127.28.207.65:42507/ using document root <none> and password file <none>
I20260929 18:36:25.194203 14682 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:25.194275 14682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:25.194504 14682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:25.197585 14682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-0/data/instance:
uuid: "321168affa4e4ef7be0b469966d96763"
format_stamp: "Formatted at 2026-09-29 18:36:25 on dist-test-slave-qcpv"
server_key: "23c02d800212056d8c1dbe5efba37858"
server_key_iv: "e9cc2b0a1c04b5fd5797920afe3d332d"
server_key_version: "encryptionkey@0"
I20260929 18:36:25.198216 14682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance:
uuid: "321168affa4e4ef7be0b469966d96763"
format_stamp: "Formatted at 2026-09-29 18:36:25 on dist-test-slave-qcpv"
server_key: "23c02d800212056d8c1dbe5efba37858"
server_key_iv: "e9cc2b0a1c04b5fd5797920afe3d332d"
server_key_version: "encryptionkey@0"
I20260929 18:36:25.203441 14682 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 18:36:25.206656 14696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:25.208439 14682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:36:25.208622 14682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-0/wal
uuid: "321168affa4e4ef7be0b469966d96763"
format_stamp: "Formatted at 2026-09-29 18:36:25 on dist-test-slave-qcpv"
server_key: "23c02d800212056d8c1dbe5efba37858"
server_key_iv: "e9cc2b0a1c04b5fd5797920afe3d332d"
server_key_version: "encryptionkey@0"
I20260929 18:36:25.208773 14682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:25.229254 14682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:25.230226 14682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:25.230474 14682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:25.231343 14682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:25.232559 14682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:25.232640 14682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:25.232722 14682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:25.232775 14682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:25.245918 14682 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.65:46063
I20260929 18:36:25.245939 14809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.65:46063 every 8 connection(s)
I20260929 18:36:25.247211 14682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
I20260929 18:36:25.251591 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 14682
I20260929 18:36:25.251722 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance
I20260929 18:36:25.252110 29501 external_mini_cluster.cc:1428] Setting key 09ea07aa28382f47a6379474d1895272
I20260929 18:36:25.254967 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.66:0
--local_ip_for_outbound_sockets=127.28.207.66
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:45729
--builtin_ntp_servers=127.28.207.84:39345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 18:36:25.260560 14661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:25.249438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:34811 (local address 127.28.207.126:45729)
0929 18:36:25.249685 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:25.249691 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:25.251225 (+ 1534us) server_negotiation.cc:407] Connection header received
0929 18:36:25.253092 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:25.253095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:25.253192 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:25.253284 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:25.255104 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.255741 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.257645 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.258064 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.258209 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:25.259684 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:25.259691 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:25.259698 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:25.259718 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:25.259750 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:25.259764 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:25.259861 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:25.260232 (+ 371us) server_negotiation.cc:299] Negotiation successful
0929 18:36:25.260323 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260929 18:36:25.261555 14812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:25.250054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45729 (local address 127.28.207.65:34811)
0929 18:36:25.250989 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:25.251046 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:25.252693 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:25.253599 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:25.253610 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:25.253880 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:25.254825 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:25.254888 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.255915 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.255922 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:25.257444 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:25.257455 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.258282 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.258285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:25.258500 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:25.259338 (+ 838us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:36:25.259340 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:25.259394 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:36:25.259480 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:25.259913 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:25.259931 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:25.259983 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 18:36:25.260150 (+ 167us) client_negotiation.cc:240] Negotiation successful
0929 18:36:25.260451 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":157,"threads_started":1}
I20260929 18:36:25.263365 14810 heartbeater.cc:347] Connected to a master server at 127.28.207.126:45729
I20260929 18:36:25.263785 14810 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:25.264715 14810 heartbeater.cc:511] Master 127.28.207.126:45729 requested a full tablet report, sending...
I20260929 18:36:25.267143 14618 ts_manager.cc:194] Registered new tserver with Master: 321168affa4e4ef7be0b469966d96763 (127.28.207.65:46063)
I20260929 18:36:25.268630 14618 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.207.65:34811
W20260929 18:36:25.379628 14813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:25.379946 14813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:25.379987 14813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:25.384248 14813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:25.384327 14813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:25.384387 14813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.66
I20260929 18:36:25.389549 14813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:39345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:45729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.66
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:25.390704 14813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:25.392284 14813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:25.403093 14818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:25.403093 14819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:25.403121 14821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:25.403569 14813 server_base.cc:1034] running on GCE node
I20260929 18:36:25.404197 14813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:25.405193 14813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:25.406421 14813 hybrid_clock.cc:648] HybridClock initialized: now 1790706985406347 us; error 102 us; skew 500 ppm
I20260929 18:36:25.408968 14813 webserver.cc:460] Webserver started at http://127.28.207.66:44739/ using document root <none> and password file <none>
I20260929 18:36:25.409799 14813 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:25.409900 14813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:25.410157 14813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:25.412858 14813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-1/data/instance:
uuid: "187f9cf1b69a4765918ec329141758c4"
format_stamp: "Formatted at 2026-09-29 18:36:25 on dist-test-slave-qcpv"
server_key: "2570e6e48f262d4253b9e1343891c3ff"
server_key_iv: "3402f2164a26001a29c9185c5fb2d8e0"
server_key_version: "encryptionkey@0"
I20260929 18:36:25.413584 14813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance:
uuid: "187f9cf1b69a4765918ec329141758c4"
format_stamp: "Formatted at 2026-09-29 18:36:25 on dist-test-slave-qcpv"
server_key: "2570e6e48f262d4253b9e1343891c3ff"
server_key_iv: "3402f2164a26001a29c9185c5fb2d8e0"
server_key_version: "encryptionkey@0"
I20260929 18:36:25.418629 14813 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 18:36:25.422403 14827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:25.425239 14813 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:36:25.425452 14813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-1/wal
uuid: "187f9cf1b69a4765918ec329141758c4"
format_stamp: "Formatted at 2026-09-29 18:36:25 on dist-test-slave-qcpv"
server_key: "2570e6e48f262d4253b9e1343891c3ff"
server_key_iv: "3402f2164a26001a29c9185c5fb2d8e0"
server_key_version: "encryptionkey@0"
I20260929 18:36:25.425625 14813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:25.443684 14813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:25.444765 14813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:25.445005 14813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:25.445755 14813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:25.447057 14813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:25.447130 14813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:25.447213 14813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:25.447258 14813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:25.461021 14813 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.66:45597
I20260929 18:36:25.461061 14940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.66:45597 every 8 connection(s)
I20260929 18:36:25.462184 14813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
I20260929 18:36:25.465081 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 14813
I20260929 18:36:25.465235 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance
I20260929 18:36:25.465618 29501 external_mini_cluster.cc:1428] Setting key 0f5acccea50c07687993cb1e12bbe9d5
I20260929 18:36:25.468746 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.67:0
--local_ip_for_outbound_sockets=127.28.207.67
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:45729
--builtin_ntp_servers=127.28.207.84:39345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 18:36:25.475543 14661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:25.464696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:57061 (local address 127.28.207.126:45729)
0929 18:36:25.465073 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:25.465079 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:25.466836 (+ 1757us) server_negotiation.cc:407] Connection header received
0929 18:36:25.468703 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:25.468706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:25.468766 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:25.468914 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:25.470269 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.471257 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.472532 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.472954 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.473090 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:25.474605 (+ 1515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:25.474611 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:25.474617 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:25.474641 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:25.474677 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:25.474691 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:25.474784 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:25.475163 (+ 379us) server_negotiation.cc:299] Negotiation successful
0929 18:36:25.475277 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192}
I20260929 18:36:25.476203 14943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:25.465541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45729 (local address 127.28.207.66:57061)
0929 18:36:25.466641 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:25.466712 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:25.468417 (+ 1705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:25.469142 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:25.469155 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:25.469281 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:25.470077 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:25.470099 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.471422 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.471425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:25.472392 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:25.472402 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.473142 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.473145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:25.473311 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:25.474245 (+ 934us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:36:25.474247 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:25.474289 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:36:25.474384 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:25.474818 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:25.474826 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:25.474831 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:36:25.475033 (+ 202us) client_negotiation.cc:240] Negotiation successful
0929 18:36:25.475282 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":168,"threads_started":1}
I20260929 18:36:25.478354 14941 heartbeater.cc:347] Connected to a master server at 127.28.207.126:45729
I20260929 18:36:25.479064 14941 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:25.479871 14941 heartbeater.cc:511] Master 127.28.207.126:45729 requested a full tablet report, sending...
I20260929 18:36:25.481484 14618 ts_manager.cc:194] Registered new tserver with Master: 187f9cf1b69a4765918ec329141758c4 (127.28.207.66:45597)
I20260929 18:36:25.482208 14618 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.207.66:57061
W20260929 18:36:25.613353 14944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:25.613734 14944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:25.613826 14944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:25.620381 14944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:25.620488 14944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:25.620558 14944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.67
I20260929 18:36:25.627126 14944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:39345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:45729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.67
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:25.628700 14944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:25.630543 14944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:25.643103 14950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:25.643009 14949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:25.643050 14952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:25.643075 14944 server_base.cc:1034] running on GCE node
I20260929 18:36:25.644239 14944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:25.645255 14944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:25.646581 14944 hybrid_clock.cc:648] HybridClock initialized: now 1790706985646558 us; error 40 us; skew 500 ppm
I20260929 18:36:25.649322 14944 webserver.cc:460] Webserver started at http://127.28.207.67:33407/ using document root <none> and password file <none>
I20260929 18:36:25.650087 14944 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:25.650203 14944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:25.650782 14944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:25.653656 14944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-2/data/instance:
uuid: "310198b87e444f6e8a0c183c2010ff20"
format_stamp: "Formatted at 2026-09-29 18:36:25 on dist-test-slave-qcpv"
server_key: "93165a89589f8d214c36311b5dd799d4"
server_key_iv: "3a24c33b581d220a1aefec40cb6c6ac8"
server_key_version: "encryptionkey@0"
I20260929 18:36:25.654428 14944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance:
uuid: "310198b87e444f6e8a0c183c2010ff20"
format_stamp: "Formatted at 2026-09-29 18:36:25 on dist-test-slave-qcpv"
server_key: "93165a89589f8d214c36311b5dd799d4"
server_key_iv: "3a24c33b581d220a1aefec40cb6c6ac8"
server_key_version: "encryptionkey@0"
I20260929 18:36:25.660231 14944 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 18:36:25.663888 14958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:25.665850 14944 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:36:25.665987 14944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-2/wal
uuid: "310198b87e444f6e8a0c183c2010ff20"
format_stamp: "Formatted at 2026-09-29 18:36:25 on dist-test-slave-qcpv"
server_key: "93165a89589f8d214c36311b5dd799d4"
server_key_iv: "3a24c33b581d220a1aefec40cb6c6ac8"
server_key_version: "encryptionkey@0"
I20260929 18:36:25.666178 14944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:25.682848 14944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:25.683669 14944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:25.683876 14944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:25.684499 14944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:25.685755 14944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:25.685856 14944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:25.685946 14944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:25.686014 14944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:25.697924 14944 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.67:34651
I20260929 18:36:25.697985 15071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.67:34651 every 8 connection(s)
I20260929 18:36:25.699263 14944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
I20260929 18:36:25.709131 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 14944
I20260929 18:36:25.709327 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance
I20260929 18:36:25.709728 29501 external_mini_cluster.cc:1428] Setting key b93c70a372b5a70b661c1b3177fdb3fe
I20260929 18:36:25.711805 14661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:25.702004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:40257 (local address 127.28.207.126:45729)
0929 18:36:25.702205 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:25.702209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:25.703775 (+ 1566us) server_negotiation.cc:407] Connection header received
0929 18:36:25.705455 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:25.705459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:25.705534 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:25.705668 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:25.706737 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.707392 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.708715 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.708988 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.709125 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:25.711095 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:25.711101 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:25.711104 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:25.711123 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:25.711153 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:25.711165 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:25.711218 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:25.711512 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 18:36:25.711596 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 18:36:25.713186 15074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:25.702500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:45729 (local address 127.28.207.67:40257)
0929 18:36:25.703569 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:25.703633 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:25.705190 (+ 1557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:25.705894 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:25.705902 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:25.706000 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:25.706568 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:25.706583 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.707551 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.707560 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:25.708562 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:25.708574 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.709245 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.709253 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:25.709533 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:25.710590 (+ 1057us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:36:25.710593 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:25.710683 (+ 90us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:36:25.710811 (+ 128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:25.711319 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:25.711329 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:25.711334 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:36:25.711457 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 18:36:25.711860 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":183,"threads_started":1}
I20260929 18:36:25.715132 15072 heartbeater.cc:347] Connected to a master server at 127.28.207.126:45729
I20260929 18:36:25.715500 15072 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:25.716228 15072 heartbeater.cc:511] Master 127.28.207.126:45729 requested a full tablet report, sending...
I20260929 18:36:25.717643 14618 ts_manager.cc:194] Registered new tserver with Master: 310198b87e444f6e8a0c183c2010ff20 (127.28.207.67:34651)
I20260929 18:36:25.718254 14618 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.207.67:40257
I20260929 18:36:25.725481 29501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:36:25.733824 14661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:25.728184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49564 (local address 127.28.207.126:45729)
0929 18:36:25.728378 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:25.728383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:25.728716 (+ 333us) server_negotiation.cc:407] Connection header received
0929 18:36:25.728923 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:25.728926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:25.728985 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:25.729094 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:25.729497 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.730187 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.731722 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.731971 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.732079 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:25.733227 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:25.733231 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:25.733233 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:25.733248 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:25.733273 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:25.733281 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:25.733360 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:25.733584 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 18:36:25.733661 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 18:36:25.738628 14618 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:36:25.741967 14618 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:36:25.761413 15086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:25.754270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56678 (local address 127.28.207.65:46063)
0929 18:36:25.755127 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:25.755137 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:25.755186 (+ 49us) server_negotiation.cc:407] Connection header received
0929 18:36:25.755319 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:25.755329 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:25.755802 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:25.756022 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:25.756469 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.757843 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.759766 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.760545 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.760818 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:25.761049 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:25.761108 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 18:36:25.761234 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":376,"threads_started":1}
I20260929 18:36:25.761415 15083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:25.753546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:46063 (local address 127.0.0.1:56678)
0929 18:36:25.754067 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:25.754116 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:25.754260 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:25.756007 (+ 1747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:25.756014 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:25.756041 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:25.756235 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:25.756242 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.758198 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.758202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:25.759606 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:25.759613 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.760693 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.760697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:25.760867 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:25.760915 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 18:36:25.761037 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 18:36:25.761207 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":276,"thread_start_us":86,"threads_started":1}
I20260929 18:36:25.762377 15084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:25.754297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:45597 (local address 127.0.0.1:60076)
0929 18:36:25.755248 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:25.755264 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:25.755567 (+ 303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:25.756225 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:25.756228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:25.756261 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:25.756484 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:25.756489 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.757747 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.757750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:25.759270 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:25.759281 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.760671 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.760674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:25.761841 (+ 1167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:25.761909 (+ 68us) client_negotiation.cc:769] Sending connection context
0929 18:36:25.761982 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 18:36:25.762215 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840,"mutex_wait_us":308,"spinlock_wait_cycles":6528,"thread_start_us":114,"threads_started":1}
I20260929 18:36:25.762554 15085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:25.753848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:34651 (local address 127.0.0.1:38572)
0929 18:36:25.754533 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:25.754548 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:25.754640 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:25.756225 (+ 1585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:25.756228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:25.756262 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:25.756476 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:25.756485 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.758182 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.758187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:25.759875 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:25.759887 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.761732 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.761736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:25.761844 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:25.761894 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 18:36:25.761982 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 18:36:25.762320 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"mutex_wait_us":61,"spinlock_wait_cycles":7168,"thread_start_us":152,"threads_started":1}
I20260929 18:36:25.762441 15087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:25.754752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60076 (local address 127.28.207.66:45597)
0929 18:36:25.755387 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:25.755399 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:25.755426 (+ 27us) server_negotiation.cc:407] Connection header received
0929 18:36:25.755748 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:25.755753 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:25.755920 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:25.756075 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:25.756626 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.757481 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.759467 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.760527 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.760669 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:25.760941 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:25.762091 (+ 1150us) server_negotiation.cc:299] Negotiation successful
0929 18:36:25.762238 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":197,"threads_started":1}
I20260929 18:36:25.763271 15088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:25.755251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38572 (local address 127.28.207.67:34651)
0929 18:36:25.755594 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:25.755602 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:25.755620 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:36:25.755714 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:25.755719 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:25.755920 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:25.756075 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:25.756639 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.757837 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.760196 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.761608 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.761728 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:25.761896 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:25.763009 (+ 1113us) server_negotiation.cc:299] Negotiation successful
0929 18:36:25.763108 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":93,"threads_started":1}
I20260929 18:36:25.764950 14744 tablet_service.cc:1467] Processing CreateTablet for tablet 3f899739515040e0b57885b4477452f0 (DEFAULT_TABLE table=test-table [id=08558617ac9f4c6ca2e1ad5efac27d59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:25.766153 14744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f899739515040e0b57885b4477452f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:25.766047 14875 tablet_service.cc:1467] Processing CreateTablet for tablet 3f899739515040e0b57885b4477452f0 (DEFAULT_TABLE table=test-table [id=08558617ac9f4c6ca2e1ad5efac27d59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:25.766047 15006 tablet_service.cc:1467] Processing CreateTablet for tablet 3f899739515040e0b57885b4477452f0 (DEFAULT_TABLE table=test-table [id=08558617ac9f4c6ca2e1ad5efac27d59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:25.767297 14875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f899739515040e0b57885b4477452f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:25.767297 15006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f899739515040e0b57885b4477452f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:25.772624 15089 tablet_bootstrap.cc:492] T 3f899739515040e0b57885b4477452f0 P 321168affa4e4ef7be0b469966d96763: Bootstrap starting.
I20260929 18:36:25.774992 15089 tablet_bootstrap.cc:654] T 3f899739515040e0b57885b4477452f0 P 321168affa4e4ef7be0b469966d96763: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:25.775969 15089 log.cc:824] T 3f899739515040e0b57885b4477452f0 P 321168affa4e4ef7be0b469966d96763: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:25.777413 15091 tablet_bootstrap.cc:492] T 3f899739515040e0b57885b4477452f0 P 187f9cf1b69a4765918ec329141758c4: Bootstrap starting.
I20260929 18:36:25.778223 15089 tablet_bootstrap.cc:492] T 3f899739515040e0b57885b4477452f0 P 321168affa4e4ef7be0b469966d96763: No bootstrap required, opened a new log
I20260929 18:36:25.778429 15089 ts_tablet_manager.cc:1397] T 3f899739515040e0b57885b4477452f0 P 321168affa4e4ef7be0b469966d96763: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260929 18:36:25.778581 15092 tablet_bootstrap.cc:492] T 3f899739515040e0b57885b4477452f0 P 310198b87e444f6e8a0c183c2010ff20: Bootstrap starting.
I20260929 18:36:25.780668 15091 tablet_bootstrap.cc:654] T 3f899739515040e0b57885b4477452f0 P 187f9cf1b69a4765918ec329141758c4: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:25.781028 15092 tablet_bootstrap.cc:654] T 3f899739515040e0b57885b4477452f0 P 310198b87e444f6e8a0c183c2010ff20: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:25.782007 15091 log.cc:824] T 3f899739515040e0b57885b4477452f0 P 187f9cf1b69a4765918ec329141758c4: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:25.782013 15092 log.cc:824] T 3f899739515040e0b57885b4477452f0 P 310198b87e444f6e8a0c183c2010ff20: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:25.783777 15089 raft_consensus.cc:348] T 3f899739515040e0b57885b4477452f0 P 321168affa4e4ef7be0b469966d96763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "321168affa4e4ef7be0b469966d96763" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46063 } } peers { permanent_uuid: "310198b87e444f6e8a0c183c2010ff20" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34651 } } peers { permanent_uuid: "187f9cf1b69a4765918ec329141758c4" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 45597 } }
I20260929 18:36:25.784042 15089 raft_consensus.cc:374] T 3f899739515040e0b57885b4477452f0 P 321168affa4e4ef7be0b469966d96763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:25.784098 15089 raft_consensus.cc:729] T 3f899739515040e0b57885b4477452f0 P 321168affa4e4ef7be0b469966d96763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 321168affa4e4ef7be0b469966d96763, State: Initialized, Role: FOLLOWER
I20260929 18:36:25.784754 15092 tablet_bootstrap.cc:492] T 3f899739515040e0b57885b4477452f0 P 310198b87e444f6e8a0c183c2010ff20: No bootstrap required, opened a new log
I20260929 18:36:25.784773 15091 tablet_bootstrap.cc:492] T 3f899739515040e0b57885b4477452f0 P 187f9cf1b69a4765918ec329141758c4: No bootstrap required, opened a new log
I20260929 18:36:25.784741 15089 consensus_queue.cc:260] T 3f899739515040e0b57885b4477452f0 P 321168affa4e4ef7be0b469966d96763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "321168affa4e4ef7be0b469966d96763" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46063 } } peers { permanent_uuid: "310198b87e444f6e8a0c183c2010ff20" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34651 } } peers { permanent_uuid: "187f9cf1b69a4765918ec329141758c4" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 45597 } }
I20260929 18:36:25.785017 15091 ts_tablet_manager.cc:1397] T 3f899739515040e0b57885b4477452f0 P 187f9cf1b69a4765918ec329141758c4: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 18:36:25.785017 15092 ts_tablet_manager.cc:1397] T 3f899739515040e0b57885b4477452f0 P 310198b87e444f6e8a0c183c2010ff20: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 18:36:25.785660 14810 heartbeater.cc:503] Master 127.28.207.126:45729 was elected leader, sending a full tablet report...
I20260929 18:36:25.786150 15089 ts_tablet_manager.cc:1428] T 3f899739515040e0b57885b4477452f0 P 321168affa4e4ef7be0b469966d96763: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20260929 18:36:25.790222 15092 raft_consensus.cc:348] T 3f899739515040e0b57885b4477452f0 P 310198b87e444f6e8a0c183c2010ff20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "321168affa4e4ef7be0b469966d96763" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46063 } } peers { permanent_uuid: "310198b87e444f6e8a0c183c2010ff20" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34651 } } peers { permanent_uuid: "187f9cf1b69a4765918ec329141758c4" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 45597 } }
I20260929 18:36:25.790222 15091 raft_consensus.cc:348] T 3f899739515040e0b57885b4477452f0 P 187f9cf1b69a4765918ec329141758c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "321168affa4e4ef7be0b469966d96763" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46063 } } peers { permanent_uuid: "310198b87e444f6e8a0c183c2010ff20" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34651 } } peers { permanent_uuid: "187f9cf1b69a4765918ec329141758c4" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 45597 } }
I20260929 18:36:25.790587 15091 raft_consensus.cc:374] T 3f899739515040e0b57885b4477452f0 P 187f9cf1b69a4765918ec329141758c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:25.790587 15092 raft_consensus.cc:374] T 3f899739515040e0b57885b4477452f0 P 310198b87e444f6e8a0c183c2010ff20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:25.790650 15092 raft_consensus.cc:729] T 3f899739515040e0b57885b4477452f0 P 310198b87e444f6e8a0c183c2010ff20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 310198b87e444f6e8a0c183c2010ff20, State: Initialized, Role: FOLLOWER
I20260929 18:36:25.790650 15091 raft_consensus.cc:729] T 3f899739515040e0b57885b4477452f0 P 187f9cf1b69a4765918ec329141758c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 187f9cf1b69a4765918ec329141758c4, State: Initialized, Role: FOLLOWER
I20260929 18:36:25.791256 15092 consensus_queue.cc:260] T 3f899739515040e0b57885b4477452f0 P 310198b87e444f6e8a0c183c2010ff20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "321168affa4e4ef7be0b469966d96763" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46063 } } peers { permanent_uuid: "310198b87e444f6e8a0c183c2010ff20" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34651 } } peers { permanent_uuid: "187f9cf1b69a4765918ec329141758c4" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 45597 } }
I20260929 18:36:25.791251 15091 consensus_queue.cc:260] T 3f899739515040e0b57885b4477452f0 P 187f9cf1b69a4765918ec329141758c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "321168affa4e4ef7be0b469966d96763" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46063 } } peers { permanent_uuid: "310198b87e444f6e8a0c183c2010ff20" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34651 } } peers { permanent_uuid: "187f9cf1b69a4765918ec329141758c4" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 45597 } }
I20260929 18:36:25.792518 15092 ts_tablet_manager.cc:1428] T 3f899739515040e0b57885b4477452f0 P 310198b87e444f6e8a0c183c2010ff20: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
I20260929 18:36:25.792526 15072 heartbeater.cc:503] Master 127.28.207.126:45729 was elected leader, sending a full tablet report...
I20260929 18:36:25.792518 15091 ts_tablet_manager.cc:1428] T 3f899739515040e0b57885b4477452f0 P 187f9cf1b69a4765918ec329141758c4: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260929 18:36:25.792539 14941 heartbeater.cc:503] Master 127.28.207.126:45729 was elected leader, sending a full tablet report...
I20260929 18:36:25.847095 15096 raft_consensus.cc:482] T 3f899739515040e0b57885b4477452f0 P 187f9cf1b69a4765918ec329141758c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:36:25.847319 15096 raft_consensus.cc:504] T 3f899739515040e0b57885b4477452f0 P 187f9cf1b69a4765918ec329141758c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "321168affa4e4ef7be0b469966d96763" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46063 } } peers { permanent_uuid: "310198b87e444f6e8a0c183c2010ff20" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34651 } } peers { permanent_uuid: "187f9cf1b69a4765918ec329141758c4" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 45597 } }
I20260929 18:36:25.848644 15096 leader_election.cc:288] T 3f899739515040e0b57885b4477452f0 P 187f9cf1b69a4765918ec329141758c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 321168affa4e4ef7be0b469966d96763 (127.28.207.65:46063), 310198b87e444f6e8a0c183c2010ff20 (127.28.207.67:34651)
I20260929 18:36:25.854493 15088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:25.849251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:38295 (local address 127.28.207.67:34651)
0929 18:36:25.849383 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:25.849387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:25.849857 (+ 470us) server_negotiation.cc:407] Connection header received
0929 18:36:25.849984 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:25.849987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:25.850051 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:25.850150 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:25.850400 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.851184 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.852960 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.853618 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.853762 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:25.853829 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:25.854233 (+ 404us) server_negotiation.cc:299] Negotiation successful
0929 18:36:25.854325 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:36:25.854493 15086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:25.849128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:55291 (local address 127.28.207.65:46063)
0929 18:36:25.849346 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:25.849353 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:25.849378 (+ 25us) server_negotiation.cc:407] Connection header received
0929 18:36:25.849467 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:25.849471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:25.849545 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:25.849638 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:25.850021 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.850955 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.852960 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.853619 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.853762 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:25.853824 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:25.854233 (+ 409us) server_negotiation.cc:299] Negotiation successful
0929 18:36:25.854332 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260929 18:36:25.854511 14943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:25.848925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:46063 (local address 127.28.207.66:55291)
0929 18:36:25.849075 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:25.849094 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:25.849211 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:25.849687 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:25.849691 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:25.849714 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:25.849868 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:25.849878 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.851124 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.851127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:25.852758 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:25.852772 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.853794 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.853797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:25.854018 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:25.854080 (+ 62us) client_negotiation.cc:769] Sending connection context
0929 18:36:25.854169 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 18:36:25.854294 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":162}
I20260929 18:36:25.854518 15098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:25.849152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:34651 (local address 127.28.207.66:38295)
0929 18:36:25.849687 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:25.849701 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:25.849803 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:25.850167 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:25.850170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:25.850185 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:25.850274 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:25.850278 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.851315 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.851318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:25.852768 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:25.852774 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.853788 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.853794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:25.853970 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:25.854070 (+ 100us) client_negotiation.cc:769] Sending connection context
0929 18:36:25.854169 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 18:36:25.854294 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":129,"thread_start_us":201,"threads_started":1}
I20260929 18:36:25.855388 14764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f899739515040e0b57885b4477452f0" candidate_uuid: "187f9cf1b69a4765918ec329141758c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "321168affa4e4ef7be0b469966d96763" is_pre_election: true
I20260929 18:36:25.855803 14764 raft_consensus.cc:2454] T 3f899739515040e0b57885b4477452f0 P 321168affa4e4ef7be0b469966d96763 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 187f9cf1b69a4765918ec329141758c4 in term 0.
I20260929 18:36:25.856467 14830 leader_election.cc:302] T 3f899739515040e0b57885b4477452f0 P 187f9cf1b69a4765918ec329141758c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 187f9cf1b69a4765918ec329141758c4, 321168affa4e4ef7be0b469966d96763; no voters:
I20260929 18:36:25.856911 15096 raft_consensus.cc:2781] T 3f899739515040e0b57885b4477452f0 P 187f9cf1b69a4765918ec329141758c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:36:25.856895 15026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f899739515040e0b57885b4477452f0" candidate_uuid: "187f9cf1b69a4765918ec329141758c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "310198b87e444f6e8a0c183c2010ff20" is_pre_election: true
I20260929 18:36:25.857024 15096 raft_consensus.cc:482] T 3f899739515040e0b57885b4477452f0 P 187f9cf1b69a4765918ec329141758c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:36:25.857059 15096 raft_consensus.cc:3037] T 3f899739515040e0b57885b4477452f0 P 187f9cf1b69a4765918ec329141758c4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:25.857244 15026 raft_consensus.cc:2454] T 3f899739515040e0b57885b4477452f0 P 310198b87e444f6e8a0c183c2010ff20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 187f9cf1b69a4765918ec329141758c4 in term 0.
I20260929 18:36:25.858757 15096 raft_consensus.cc:504] T 3f899739515040e0b57885b4477452f0 P 187f9cf1b69a4765918ec329141758c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "321168affa4e4ef7be0b469966d96763" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46063 } } peers { permanent_uuid: "310198b87e444f6e8a0c183c2010ff20" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34651 } } peers { permanent_uuid: "187f9cf1b69a4765918ec329141758c4" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 45597 } }
I20260929 18:36:25.859292 15096 leader_election.cc:288] T 3f899739515040e0b57885b4477452f0 P 187f9cf1b69a4765918ec329141758c4 [CANDIDATE]: Term 1 election: Requested vote from peers 321168affa4e4ef7be0b469966d96763 (127.28.207.65:46063), 310198b87e444f6e8a0c183c2010ff20 (127.28.207.67:34651)
I20260929 18:36:25.859712 14764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f899739515040e0b57885b4477452f0" candidate_uuid: "187f9cf1b69a4765918ec329141758c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "321168affa4e4ef7be0b469966d96763"
I20260929 18:36:25.859907 14764 raft_consensus.cc:3037] T 3f899739515040e0b57885b4477452f0 P 321168affa4e4ef7be0b469966d96763 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:25.860080 15026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f899739515040e0b57885b4477452f0" candidate_uuid: "187f9cf1b69a4765918ec329141758c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "310198b87e444f6e8a0c183c2010ff20"
I20260929 18:36:25.860277 15026 raft_consensus.cc:3037] T 3f899739515040e0b57885b4477452f0 P 310198b87e444f6e8a0c183c2010ff20 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:25.861217 14764 raft_consensus.cc:2454] T 3f899739515040e0b57885b4477452f0 P 321168affa4e4ef7be0b469966d96763 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 187f9cf1b69a4765918ec329141758c4 in term 1.
I20260929 18:36:25.861574 14830 leader_election.cc:302] T 3f899739515040e0b57885b4477452f0 P 187f9cf1b69a4765918ec329141758c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 187f9cf1b69a4765918ec329141758c4, 321168affa4e4ef7be0b469966d96763; no voters:
I20260929 18:36:25.861747 15026 raft_consensus.cc:2454] T 3f899739515040e0b57885b4477452f0 P 310198b87e444f6e8a0c183c2010ff20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 187f9cf1b69a4765918ec329141758c4 in term 1.
I20260929 18:36:25.861806 15096 raft_consensus.cc:2781] T 3f899739515040e0b57885b4477452f0 P 187f9cf1b69a4765918ec329141758c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:25.862138 15096 raft_consensus.cc:686] T 3f899739515040e0b57885b4477452f0 P 187f9cf1b69a4765918ec329141758c4 [term 1 LEADER]: Becoming Leader. State: Replica: 187f9cf1b69a4765918ec329141758c4, State: Running, Role: LEADER
I20260929 18:36:25.862632 15096 consensus_queue.cc:237] T 3f899739515040e0b57885b4477452f0 P 187f9cf1b69a4765918ec329141758c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "321168affa4e4ef7be0b469966d96763" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46063 } } peers { permanent_uuid: "310198b87e444f6e8a0c183c2010ff20" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34651 } } peers { permanent_uuid: "187f9cf1b69a4765918ec329141758c4" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 45597 } }
I20260929 18:36:25.867274 14622 catalog_manager.cc:5495] T 3f899739515040e0b57885b4477452f0 P 187f9cf1b69a4765918ec329141758c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 187f9cf1b69a4765918ec329141758c4 (127.28.207.66). New cstate: current_term: 1 leader_uuid: "187f9cf1b69a4765918ec329141758c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "321168affa4e4ef7be0b469966d96763" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "310198b87e444f6e8a0c183c2010ff20" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "187f9cf1b69a4765918ec329141758c4" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 45597 } health_report { overall_health: HEALTHY } } }
I20260929 18:36:25.887400 15087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:25.883042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60090 (local address 127.28.207.66:45597)
0929 18:36:25.883208 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:25.883213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:25.883232 (+ 19us) server_negotiation.cc:407] Connection header received
0929 18:36:25.883557 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:25.883560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:25.883651 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:25.883790 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:36:25.884067 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.884641 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.885559 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:25.885789 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:25.885904 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:25.886260 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:36:25.886933 (+ 673us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:36:25.887074 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:25.887112 (+ 38us) server_negotiation.cc:299] Negotiation successful
0929 18:36:25.887181 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 18:36:25.894125 14764 raft_consensus.cc:1261] T 3f899739515040e0b57885b4477452f0 P 321168affa4e4ef7be0b469966d96763 [term 1 FOLLOWER]: Refusing update from remote peer 187f9cf1b69a4765918ec329141758c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:36:25.894331 15026 raft_consensus.cc:1261] T 3f899739515040e0b57885b4477452f0 P 310198b87e444f6e8a0c183c2010ff20 [term 1 FOLLOWER]: Refusing update from remote peer 187f9cf1b69a4765918ec329141758c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:36:25.894958 15096 consensus_queue.cc:1035] T 3f899739515040e0b57885b4477452f0 P 187f9cf1b69a4765918ec329141758c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "321168affa4e4ef7be0b469966d96763" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 46063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:36:25.895490 15099 consensus_queue.cc:1035] T 3f899739515040e0b57885b4477452f0 P 187f9cf1b69a4765918ec329141758c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "310198b87e444f6e8a0c183c2010ff20" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 34651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:36:25.905002 15103 mvcc.cc:205] Tried to move back new op lower bound from 7334735814213341184 to 7334735814096977920. Current Snapshot: MvccSnapshot[applied={T|T < 7334735814213341184}]
I20260929 18:36:25.905009 15104 mvcc.cc:205] Tried to move back new op lower bound from 7334735814213341184 to 7334735814096977920. Current Snapshot: MvccSnapshot[applied={T|T < 7334735814213341184}]
I20260929 18:36:25.915069 14618 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:36:25.915300 14618 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:36:25.918699 14618 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0283f35103594cecbd48c1ae80fbaeed: Sending DeleteTablet for 3 replicas of tablet 3f899739515040e0b57885b4477452f0
I20260929 18:36:25.920068 15006 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f899739515040e0b57885b4477452f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:25 UTC) from {username='slave'} at 127.0.0.1:38572
I20260929 18:36:25.920948 15112 tablet_replica.cc:330] T 3f899739515040e0b57885b4477452f0 P 310198b87e444f6e8a0c183c2010ff20: stopping tablet replica
I20260929 18:36:25.921648 15112 raft_consensus.cc:2229] T 3f899739515040e0b57885b4477452f0 P 310198b87e444f6e8a0c183c2010ff20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:36:25.921765 14875 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f899739515040e0b57885b4477452f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:25 UTC) from {username='slave'} at 127.0.0.1:60076
I20260929 18:36:25.922183 15112 raft_consensus.cc:2258] T 3f899739515040e0b57885b4477452f0 P 310198b87e444f6e8a0c183c2010ff20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:36:25.922158 14744 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f899739515040e0b57885b4477452f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:25 UTC) from {username='slave'} at 127.0.0.1:56678
I20260929 18:36:25.922338 15113 tablet_replica.cc:330] T 3f899739515040e0b57885b4477452f0 P 187f9cf1b69a4765918ec329141758c4: stopping tablet replica
I20260929 18:36:25.923038 15113 raft_consensus.cc:2229] T 3f899739515040e0b57885b4477452f0 P 187f9cf1b69a4765918ec329141758c4 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:36:25.923545 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 14682
I20260929 18:36:25.923592 15113 raft_consensus.cc:2258] T 3f899739515040e0b57885b4477452f0 P 187f9cf1b69a4765918ec329141758c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:36:25.927167 15112 ts_tablet_manager.cc:1905] T 3f899739515040e0b57885b4477452f0 P 310198b87e444f6e8a0c183c2010ff20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:36:25.927258 15113 ts_tablet_manager.cc:1905] T 3f899739515040e0b57885b4477452f0 P 187f9cf1b69a4765918ec329141758c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:36:25.930430 15112 ts_tablet_manager.cc:1918] T 3f899739515040e0b57885b4477452f0 P 310198b87e444f6e8a0c183c2010ff20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:36:25.930631 15112 log.cc:1196] T 3f899739515040e0b57885b4477452f0 P 310198b87e444f6e8a0c183c2010ff20: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790706718260899-29501-0/minicluster-data/ts-2/wal/wals/3f899739515040e0b57885b4477452f0
I20260929 18:36:25.931068 15112 ts_tablet_manager.cc:1939] T 3f899739515040e0b57885b4477452f0 P 310198b87e444f6e8a0c183c2010ff20: Deleting consensus metadata
I20260929 18:36:25.932956 14607 catalog_manager.cc:4841] TS 310198b87e444f6e8a0c183c2010ff20 (127.28.207.67:34651): tablet 3f899739515040e0b57885b4477452f0 (table test-table [id=08558617ac9f4c6ca2e1ad5efac27d59]) successfully deleted
W20260929 18:36:25.933780 14608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.207.65:46063 (error 108)
I20260929 18:36:25.935155 15084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:25.934694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:46063 (local address 127.0.0.1:56692)
0929 18:36:25.934885 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:25.935029 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.65:46063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 18:36:25.935189 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 14813
W20260929 18:36:25.935534 14608 catalog_manager.cc:4584] TS 321168affa4e4ef7be0b469966d96763 (127.28.207.65:46063): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3f899739515040e0b57885b4477452f0: Network error: Client connection negotiation failed: client connection to 127.28.207.65:46063: connect: Connection refused (error 111)
I20260929 18:36:25.942281 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 14944
I20260929 18:36:25.944460 15084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:25.942420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:45597 (local address 127.0.0.1:60098)
0929 18:36:25.944258 (+ 1838us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:25.944343 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.66:45597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1709}
I20260929 18:36:25.949482 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 14591
2026-09-29T18:36:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1239 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:36:26 dist-test-slave-qcpv krb5kdc[15118](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:36:26 dist-test-slave-qcpv krb5kdc[15118](info): set up 2 sockets
Sep 29 18:36:26 dist-test-slave-qcpv krb5kdc[15118](info): commencing operation
krb5kdc: starting...
W20260929 18:36:27.019513 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:36:27 dist-test-slave-qcpv krb5kdc[15118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:36:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:36:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.207.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
I20260929 18:36:27.168416 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:40107
--webserver_interface=127.28.207.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:44887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:40107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:27.285995 15131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:27.286325 15131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:27.286409 15131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:27.290557 15131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:36:27.290664 15131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:27.290710 15131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:27.290751 15131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:36:27.290786 15131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:36:27.296144 15131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:44887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:40107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:40107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:27.297286 15131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:27.298342 15131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:27.308339 15137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:27.308339 15136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:27.308346 15139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:27.308822 15131 server_base.cc:1034] running on GCE node
I20260929 18:36:27.309576 15131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:27.310676 15131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:27.311895 15131 hybrid_clock.cc:648] HybridClock initialized: now 1790706987311880 us; error 45 us; skew 500 ppm
Sep 29 18:36:27 dist-test-slave-qcpv krb5kdc[15118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706987, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:27.324113 15131 init.cc:377] Logged in from keytab as kudu/127.28.207.126@KRBTEST.COM (short username kudu)
I20260929 18:36:27.325219 15131 webserver.cc:460] Webserver started at http://127.28.207.126:35649/ using document root <none> and password file <none>
I20260929 18:36:27.325835 15131 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:27.325901 15131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:27.326136 15131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:27.328325 15131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "424a6f3dc6f1432d8c37b9b493243ea7"
format_stamp: "Formatted at 2026-09-29 18:36:27 on dist-test-slave-qcpv"
server_key: "d890325b43cfbe396b8dcaada7a83fff"
server_key_iv: "043bbf4ca3774e1465858cd056c0f480"
server_key_version: "encryptionkey@0"
I20260929 18:36:27.328905 15131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "424a6f3dc6f1432d8c37b9b493243ea7"
format_stamp: "Formatted at 2026-09-29 18:36:27 on dist-test-slave-qcpv"
server_key: "d890325b43cfbe396b8dcaada7a83fff"
server_key_iv: "043bbf4ca3774e1465858cd056c0f480"
server_key_version: "encryptionkey@0"
I20260929 18:36:27.333149 15131 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:36:27.336153 15146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:27.337673 15131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 18:36:27.337847 15131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "424a6f3dc6f1432d8c37b9b493243ea7"
format_stamp: "Formatted at 2026-09-29 18:36:27 on dist-test-slave-qcpv"
server_key: "d890325b43cfbe396b8dcaada7a83fff"
server_key_iv: "043bbf4ca3774e1465858cd056c0f480"
server_key_version: "encryptionkey@0"
I20260929 18:36:27.337985 15131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:27.356033 15131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:27.359863 15131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:27.360076 15131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:27.368520 15131 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:40107
I20260929 18:36:27.368554 15198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:40107 every 8 connection(s)
I20260929 18:36:27.369514 15131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:36:27.373567 15199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:27.377741 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 15131
I20260929 18:36:27.377862 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:36:27.378198 29501 external_mini_cluster.cc:1428] Setting key f2ba187169e5941341a7e0878d8215d5
I20260929 18:36:27.382344 15199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 424a6f3dc6f1432d8c37b9b493243ea7: Bootstrap starting.
I20260929 18:36:27.386524 15199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 424a6f3dc6f1432d8c37b9b493243ea7: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:27.387705 15199 log.cc:824] T 00000000000000000000000000000000 P 424a6f3dc6f1432d8c37b9b493243ea7: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:27.390340 15199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 424a6f3dc6f1432d8c37b9b493243ea7: No bootstrap required, opened a new log
Sep 29 18:36:27 dist-test-slave-qcpv krb5kdc[15118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:36:27.395020 15199 raft_consensus.cc:348] T 00000000000000000000000000000000 P 424a6f3dc6f1432d8c37b9b493243ea7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "424a6f3dc6f1432d8c37b9b493243ea7" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 40107 } }
I20260929 18:36:27.395365 15199 raft_consensus.cc:374] T 00000000000000000000000000000000 P 424a6f3dc6f1432d8c37b9b493243ea7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:27.395431 15199 raft_consensus.cc:729] T 00000000000000000000000000000000 P 424a6f3dc6f1432d8c37b9b493243ea7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 424a6f3dc6f1432d8c37b9b493243ea7, State: Initialized, Role: FOLLOWER
I20260929 18:36:27.395965 15199 consensus_queue.cc:260] T 00000000000000000000000000000000 P 424a6f3dc6f1432d8c37b9b493243ea7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "424a6f3dc6f1432d8c37b9b493243ea7" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 40107 } }
I20260929 18:36:27.396143 15199 raft_consensus.cc:388] T 00000000000000000000000000000000 P 424a6f3dc6f1432d8c37b9b493243ea7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:36:27.396222 15199 raft_consensus.cc:482] T 00000000000000000000000000000000 P 424a6f3dc6f1432d8c37b9b493243ea7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:36:27.396330 15199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 424a6f3dc6f1432d8c37b9b493243ea7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:27.397689 15199 raft_consensus.cc:504] T 00000000000000000000000000000000 P 424a6f3dc6f1432d8c37b9b493243ea7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "424a6f3dc6f1432d8c37b9b493243ea7" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 40107 } }
I20260929 18:36:27.398061 15199 leader_election.cc:302] T 00000000000000000000000000000000 P 424a6f3dc6f1432d8c37b9b493243ea7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 424a6f3dc6f1432d8c37b9b493243ea7; no voters:
I20260929 18:36:27.398411 15199 leader_election.cc:288] T 00000000000000000000000000000000 P 424a6f3dc6f1432d8c37b9b493243ea7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:36:27.398759 15204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 424a6f3dc6f1432d8c37b9b493243ea7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:27.399034 15202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:27.379999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55814 (local address 127.28.207.126:40107)
0929 18:36:27.380591 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:27.380619 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:27.380679 (+ 60us) server_negotiation.cc:407] Connection header received
0929 18:36:27.381977 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:27.382014 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:27.382639 (+ 625us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:27.383433 (+ 794us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:27.384313 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:27.386103 (+ 1790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:27.387338 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:27.388133 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:27.388349 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:27.392112 (+ 3763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:27.392121 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:27.392140 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:27.392177 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:27.395136 (+ 2959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:27.395831 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:27.395837 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:27.395842 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:27.395993 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:27.396441 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:27.396445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:27.396447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:27.396976 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:27.397193 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:27.397591 (+ 398us) server_negotiation.cc:299] Negotiation successful
0929 18:36:27.397918 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":141,"threads_started":1}
I20260929 18:36:27.399278 15204 raft_consensus.cc:686] T 00000000000000000000000000000000 P 424a6f3dc6f1432d8c37b9b493243ea7 [term 1 LEADER]: Becoming Leader. State: Replica: 424a6f3dc6f1432d8c37b9b493243ea7, State: Running, Role: LEADER
I20260929 18:36:27.400009 15199 sys_catalog.cc:564] T 00000000000000000000000000000000 P 424a6f3dc6f1432d8c37b9b493243ea7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:36:27.400007 15204 consensus_queue.cc:237] T 00000000000000000000000000000000 P 424a6f3dc6f1432d8c37b9b493243ea7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "424a6f3dc6f1432d8c37b9b493243ea7" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 40107 } }
I20260929 18:36:27.402387 15205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 424a6f3dc6f1432d8c37b9b493243ea7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "424a6f3dc6f1432d8c37b9b493243ea7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "424a6f3dc6f1432d8c37b9b493243ea7" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 40107 } } }
I20260929 18:36:27.402663 15206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 424a6f3dc6f1432d8c37b9b493243ea7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 424a6f3dc6f1432d8c37b9b493243ea7. Latest consensus state: current_term: 1 leader_uuid: "424a6f3dc6f1432d8c37b9b493243ea7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "424a6f3dc6f1432d8c37b9b493243ea7" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 40107 } } }
I20260929 18:36:27.402691 15205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 424a6f3dc6f1432d8c37b9b493243ea7 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:27.402747 15206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 424a6f3dc6f1432d8c37b9b493243ea7 [sys.catalog]: This master's current role is: LEADER
W20260929 18:36:27.405278 15218 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 424a6f3dc6f1432d8c37b9b493243ea7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:36:27.405386 15218 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:36:27.405467 15219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:36:27.406222 15219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:36:27.412122 15219 catalog_manager.cc:1347] Generated new cluster ID: 4a74827d26f84e80913de516730f6eff
I20260929 18:36:27.412220 15219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:36:27.419432 15219 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:36:27.420470 15219 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:36:27.441355 15219 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 424a6f3dc6f1432d8c37b9b493243ea7: Generated new TSK 0
I20260929 18:36:27.442373 15219 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.207.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab.
Entry for principal kudu/127.28.207.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab.
I20260929 18:36:27.477924 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.65:0
--local_ip_for_outbound_sockets=127.28.207.65
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:40107
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:44887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:27.591204 15225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:27.591521 15225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:27.591599 15225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:27.595769 15225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:27.595860 15225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:27.595957 15225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.65
I20260929 18:36:27.602546 15225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:44887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:40107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.65
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:27.603811 15225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:27.604921 15225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:27.616045 15230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:27.616073 15233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:27.616528 15225 server_base.cc:1034] running on GCE node
W20260929 18:36:27.616113 15231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:27.617273 15225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:27.617952 15225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:27.619130 15225 hybrid_clock.cc:648] HybridClock initialized: now 1790706987619109 us; error 44 us; skew 500 ppm
Sep 29 18:36:27 dist-test-slave-qcpv krb5kdc[15118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706987, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:27.622802 15225 init.cc:377] Logged in from keytab as kudu/127.28.207.65@KRBTEST.COM (short username kudu)
I20260929 18:36:27.623925 15225 webserver.cc:460] Webserver started at http://127.28.207.65:33101/ using document root <none> and password file <none>
I20260929 18:36:27.624569 15225 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:27.624643 15225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:27.624876 15225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:27.627331 15225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-0/data/instance:
uuid: "b31ca3b0cf384a3192857eac456336bd"
format_stamp: "Formatted at 2026-09-29 18:36:27 on dist-test-slave-qcpv"
server_key: "75b296996f2df781b2304714af55b5ca"
server_key_iv: "f675106a196c33d92b0d24e52e128ffa"
server_key_version: "encryptionkey@0"
I20260929 18:36:27.627914 15225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance:
uuid: "b31ca3b0cf384a3192857eac456336bd"
format_stamp: "Formatted at 2026-09-29 18:36:27 on dist-test-slave-qcpv"
server_key: "75b296996f2df781b2304714af55b5ca"
server_key_iv: "f675106a196c33d92b0d24e52e128ffa"
server_key_version: "encryptionkey@0"
I20260929 18:36:27.632396 15225 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:36:27.635689 15240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:27.637416 15225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:36:27.637619 15225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-0/wal
uuid: "b31ca3b0cf384a3192857eac456336bd"
format_stamp: "Formatted at 2026-09-29 18:36:27 on dist-test-slave-qcpv"
server_key: "75b296996f2df781b2304714af55b5ca"
server_key_iv: "f675106a196c33d92b0d24e52e128ffa"
server_key_version: "encryptionkey@0"
I20260929 18:36:27.637758 15225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:27.657069 15225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:27.660817 15225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:27.661063 15225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:27.661726 15225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:27.662614 15225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:27.662662 15225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:27.662708 15225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:27.662747 15225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:27.673568 15225 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.65:37863
I20260929 18:36:27.673600 15353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.65:37863 every 8 connection(s)
I20260929 18:36:27.674575 15225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
I20260929 18:36:27.676653 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 15225
I20260929 18:36:27.676779 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance
I20260929 18:36:27.677151 29501 external_mini_cluster.cc:1428] Setting key 5f98bcb34507ddab981a6d3e857f9fe0
Sep 29 18:36:27 dist-test-slave-qcpv krb5kdc[15118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706987, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.207.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.66@KRBTEST.COM" created.
I20260929 18:36:27.694424 15202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:27.677536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:51889 (local address 127.28.207.126:40107)
0929 18:36:27.677751 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:27.677757 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:27.678736 (+ 979us) server_negotiation.cc:407] Connection header received
0929 18:36:27.680944 (+ 2208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:27.680951 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:27.681050 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:27.681186 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:27.682591 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:27.683536 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:27.684943 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:27.685230 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:27.685361 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:27.688865 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:27.688869 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:27.688872 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:27.688899 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:27.690707 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:27.691368 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:27.691374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:27.691376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:27.691480 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:27.692280 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:27.692284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:27.692285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:27.692576 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:27.692703 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:27.693943 (+ 1240us) server_negotiation.cc:299] Negotiation successful
0929 18:36:27.694146 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 18:36:27.695524 15356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:27.677802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:40107 (local address 127.28.207.65:51889)
0929 18:36:27.678516 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:27.678599 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:27.680415 (+ 1816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:27.681440 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:27.681449 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:27.681845 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:27.682396 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:27.682410 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:27.683740 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:27.683745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:27.684758 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:27.684769 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:27.685411 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:27.685416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:27.685659 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:27.686475 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:27.686491 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:27.688649 (+ 2158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:27.690849 (+ 2200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:27.690881 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:27.690910 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:27.691244 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:27.691669 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:27.691674 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:27.691676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:27.691957 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:27.692771 (+ 814us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:27.692779 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:27.693542 (+ 763us) client_negotiation.cc:769] Sending connection context
0929 18:36:27.693827 (+ 285us) client_negotiation.cc:240] Negotiation successful
0929 18:36:27.694229 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":204,"threads_started":1}
I20260929 18:36:27.697362 15354 heartbeater.cc:347] Connected to a master server at 127.28.207.126:40107
I20260929 18:36:27.697724 15354 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:27.698714 15354 heartbeater.cc:511] Master 127.28.207.126:40107 requested a full tablet report, sending...
I20260929 18:36:27.701548 15162 ts_manager.cc:194] Registered new tserver with Master: b31ca3b0cf384a3192857eac456336bd (127.28.207.65:37863)
I20260929 18:36:27.703585 15162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.65@KRBTEST.COM'} at 127.28.207.65:51889
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab.
Entry for principal kudu/127.28.207.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab.
I20260929 18:36:27.714675 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.66:0
--local_ip_for_outbound_sockets=127.28.207.66
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:40107
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:44887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:27.860364 15359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:27.860740 15359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:27.860791 15359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:27.865345 15359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:27.865459 15359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:27.865533 15359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.66
I20260929 18:36:27.872112 15359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:44887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:40107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.66
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:27.873886 15359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:27.875602 15359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:27.888049 15364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:27.888124 15367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:27.888124 15365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:27.888664 15359 server_base.cc:1034] running on GCE node
I20260929 18:36:27.889181 15359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:27.889971 15359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:27.891182 15359 hybrid_clock.cc:648] HybridClock initialized: now 1790706987891143 us; error 63 us; skew 500 ppm
Sep 29 18:36:27 dist-test-slave-qcpv krb5kdc[15118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706987, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:27.895828 15359 init.cc:377] Logged in from keytab as kudu/127.28.207.66@KRBTEST.COM (short username kudu)
I20260929 18:36:27.897113 15359 webserver.cc:460] Webserver started at http://127.28.207.66:45879/ using document root <none> and password file <none>
I20260929 18:36:27.897837 15359 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:27.897933 15359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:27.898216 15359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:27.901144 15359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-1/data/instance:
uuid: "4b60271c86ec4a1da15b22c25aaa21b6"
format_stamp: "Formatted at 2026-09-29 18:36:27 on dist-test-slave-qcpv"
server_key: "852ceaac7b6795dbe6a946f296113ce9"
server_key_iv: "df33b31319ac3ee13742311ede2bd1eb"
server_key_version: "encryptionkey@0"
I20260929 18:36:27.901939 15359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance:
uuid: "4b60271c86ec4a1da15b22c25aaa21b6"
format_stamp: "Formatted at 2026-09-29 18:36:27 on dist-test-slave-qcpv"
server_key: "852ceaac7b6795dbe6a946f296113ce9"
server_key_iv: "df33b31319ac3ee13742311ede2bd1eb"
server_key_version: "encryptionkey@0"
I20260929 18:36:27.907437 15359 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 18:36:27.911103 15374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:27.912896 15359 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:36:27.913089 15359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-1/wal
uuid: "4b60271c86ec4a1da15b22c25aaa21b6"
format_stamp: "Formatted at 2026-09-29 18:36:27 on dist-test-slave-qcpv"
server_key: "852ceaac7b6795dbe6a946f296113ce9"
server_key_iv: "df33b31319ac3ee13742311ede2bd1eb"
server_key_version: "encryptionkey@0"
I20260929 18:36:27.913260 15359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:27.934688 15359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:27.938753 15359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:27.939381 15359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:27.940192 15359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:27.941329 15359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:27.941406 15359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:27.941496 15359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:27.941555 15359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:27.954795 15359 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.66:45153
I20260929 18:36:27.954897 15487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.66:45153 every 8 connection(s)
I20260929 18:36:27.956094 15359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
I20260929 18:36:27.964500 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 15359
I20260929 18:36:27.964648 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance
I20260929 18:36:27.964927 29501 external_mini_cluster.cc:1428] Setting key af06c086514dbff1cc836cd8bc3b16c3
Sep 29 18:36:27 dist-test-slave-qcpv krb5kdc[15118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706987, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:36:27.973899 15202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:27.958518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:38623 (local address 127.28.207.126:40107)
0929 18:36:27.958724 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:27.958728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:27.959940 (+ 1212us) server_negotiation.cc:407] Connection header received
0929 18:36:27.961405 (+ 1465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:27.961408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:27.961464 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:27.961580 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:27.962785 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:27.963390 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:27.964808 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:27.965148 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:27.965294 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:27.968696 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:27.968700 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:27.968703 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:27.968730 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:27.971132 (+ 2402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:27.971884 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:27.971890 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:27.971892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:27.971991 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:27.972410 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:27.972415 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:27.972417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:27.972660 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:27.972784 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:27.973514 (+ 730us) server_negotiation.cc:299] Negotiation successful
0929 18:36:27.973677 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
WARNING: no policy specified for kudu/127.28.207.67@KRBTEST.COM; defaulting to no policy
I20260929 18:36:27.975355 15490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:27.959126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:40107 (local address 127.28.207.66:38623)
0929 18:36:27.959747 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:27.959805 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:27.961183 (+ 1378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:27.961754 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:27.961761 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:27.962146 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:27.962643 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:27.962656 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:27.963512 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:27.963517 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:27.964592 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:27.964605 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:27.965371 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:27.965375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:27.965522 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:27.966358 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:27.966376 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:27.968460 (+ 2084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:27.971318 (+ 2858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:27.971327 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:27.971364 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:27.971758 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:27.972097 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:27.972101 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:27.972103 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:27.972288 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:27.972783 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:27.972794 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:27.973216 (+ 422us) client_negotiation.cc:769] Sending connection context
0929 18:36:27.973469 (+ 253us) client_negotiation.cc:240] Negotiation successful
0929 18:36:27.973934 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":150,"threads_started":1}
I20260929 18:36:27.977471 15488 heartbeater.cc:347] Connected to a master server at 127.28.207.126:40107
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.67@KRBTEST.COM" created.
I20260929 18:36:27.977872 15488 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:27.979238 15488 heartbeater.cc:511] Master 127.28.207.126:40107 requested a full tablet report, sending...
I20260929 18:36:27.981104 15162 ts_manager.cc:194] Registered new tserver with Master: 4b60271c86ec4a1da15b22c25aaa21b6 (127.28.207.66:45153)
I20260929 18:36:27.982072 15162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.66@KRBTEST.COM'} at 127.28.207.66:38623
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab.
Entry for principal kudu/127.28.207.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab.
I20260929 18:36:27.998667 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.67:0
--local_ip_for_outbound_sockets=127.28.207.67
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:40107
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:44887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:28.118556 15493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:28.118937 15493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:28.119023 15493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:28.123342 15493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:28.123420 15493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:28.123481 15493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.67
I20260929 18:36:28.129333 15493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:44887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:40107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.67
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:28.130578 15493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:28.131716 15493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:28.143733 15501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:28.144274 15493 server_base.cc:1034] running on GCE node
W20260929 18:36:28.144351 15498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:28.144795 15499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:28.145713 15493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:28.146812 15493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:28.148108 15493 hybrid_clock.cc:648] HybridClock initialized: now 1790706988148063 us; error 79 us; skew 500 ppm
Sep 29 18:36:28 dist-test-slave-qcpv krb5kdc[15118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706988, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:28.151654 15493 init.cc:377] Logged in from keytab as kudu/127.28.207.67@KRBTEST.COM (short username kudu)
I20260929 18:36:28.152769 15493 webserver.cc:460] Webserver started at http://127.28.207.67:42753/ using document root <none> and password file <none>
I20260929 18:36:28.153445 15493 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:28.153508 15493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:28.153712 15493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:28.156342 15493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-2/data/instance:
uuid: "cc0a416950e64b5da7d7e3fa586d0e4b"
format_stamp: "Formatted at 2026-09-29 18:36:28 on dist-test-slave-qcpv"
server_key: "d944c4709627cdbcc2302c1e3704569c"
server_key_iv: "cb5b5138767122499adf9756f671501a"
server_key_version: "encryptionkey@0"
I20260929 18:36:28.157033 15493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance:
uuid: "cc0a416950e64b5da7d7e3fa586d0e4b"
format_stamp: "Formatted at 2026-09-29 18:36:28 on dist-test-slave-qcpv"
server_key: "d944c4709627cdbcc2302c1e3704569c"
server_key_iv: "cb5b5138767122499adf9756f671501a"
server_key_version: "encryptionkey@0"
I20260929 18:36:28.162005 15493 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 18:36:28.165076 15508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:28.167070 15493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:36:28.167241 15493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-2/wal
uuid: "cc0a416950e64b5da7d7e3fa586d0e4b"
format_stamp: "Formatted at 2026-09-29 18:36:28 on dist-test-slave-qcpv"
server_key: "d944c4709627cdbcc2302c1e3704569c"
server_key_iv: "cb5b5138767122499adf9756f671501a"
server_key_version: "encryptionkey@0"
I20260929 18:36:28.167376 15493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:28.189229 15493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:28.193549 15493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:28.193764 15493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:28.194473 15493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:28.195688 15493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:28.195744 15493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:28.195793 15493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:28.195854 15493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:28.208994 15493 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.67:41317
I20260929 18:36:28.209059 15621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.67:41317 every 8 connection(s)
I20260929 18:36:28.210175 15493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
I20260929 18:36:28.218199 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 15493
I20260929 18:36:28.218423 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance
I20260929 18:36:28.218830 29501 external_mini_cluster.cc:1428] Setting key f36eee5abc0de796e81a06341d2e7cb6
Sep 29 18:36:28 dist-test-slave-qcpv krb5kdc[15118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706988, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:36:28.226542 15202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:28.212571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:51637 (local address 127.28.207.126:40107)
0929 18:36:28.212831 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:28.212837 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:28.213873 (+ 1036us) server_negotiation.cc:407] Connection header received
0929 18:36:28.215409 (+ 1536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:28.215412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:28.215470 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:28.215585 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:28.216784 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:28.217318 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:28.218611 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:28.219164 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:28.219333 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:28.222554 (+ 3221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:28.222558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:28.222560 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:28.222588 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:28.224350 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:28.224983 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:28.224988 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:28.224990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:28.225079 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:28.225412 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:28.225417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:28.225419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:28.225639 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:28.225749 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:28.226234 (+ 485us) server_negotiation.cc:299] Negotiation successful
0929 18:36:28.226362 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260929 18:36:28.227841 15624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:28.213145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:40107 (local address 127.28.207.67:51637)
0929 18:36:28.213690 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:28.213738 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:28.215178 (+ 1440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:28.215757 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:28.215764 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:28.216145 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:28.216641 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:28.216654 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:28.217441 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:28.217446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:28.218320 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:28.218329 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:28.219390 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:28.219398 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:28.219591 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:28.220285 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:28.220301 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:28.222232 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:28.224527 (+ 2295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:28.224534 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:28.224550 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:28.224868 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:28.225178 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:28.225181 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:28.225183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:28.225317 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:28.225744 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:28.225751 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:28.226002 (+ 251us) client_negotiation.cc:769] Sending connection context
0929 18:36:28.226197 (+ 195us) client_negotiation.cc:240] Negotiation successful
0929 18:36:28.226515 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":137,"threads_started":1}
I20260929 18:36:28.229354 15622 heartbeater.cc:347] Connected to a master server at 127.28.207.126:40107
I20260929 18:36:28.229669 15622 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:28.230243 15622 heartbeater.cc:511] Master 127.28.207.126:40107 requested a full tablet report, sending...
I20260929 18:36:28.231581 15162 ts_manager.cc:194] Registered new tserver with Master: cc0a416950e64b5da7d7e3fa586d0e4b (127.28.207.67:41317)
I20260929 18:36:28.232280 15162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.67@KRBTEST.COM'} at 127.28.207.67:51637
I20260929 18:36:28.234139 29501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:36:28.246846 15202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:28.236732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55822 (local address 127.28.207.126:40107)
0929 18:36:28.236925 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:28.236929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:28.237013 (+ 84us) server_negotiation.cc:407] Connection header received
0929 18:36:28.237154 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:28.237157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:28.237200 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:28.237289 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:28.238097 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:28.238674 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:28.239830 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:28.240051 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:28.240183 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:28.242725 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:28.242728 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:28.242731 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:28.242757 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:28.244561 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:28.245195 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:28.245198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:28.245199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:28.245280 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:28.245690 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:28.245692 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:28.245693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:28.245901 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:28.246027 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:28.246466 (+ 439us) server_negotiation.cc:299] Negotiation successful
0929 18:36:28.246592 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260929 18:36:28.251156 15162 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:36:28.254680 15162 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:36:28.273780 15634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:28.266007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:41317 (local address 127.0.0.1:41946)
0929 18:36:28.268255 (+ 2248us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:28.268270 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:28.268359 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:28.268913 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:28.268916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:28.268936 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:28.269022 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:28.269028 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:28.270412 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:28.270415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:28.271730 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:28.271741 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:28.273111 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:28.273115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:28.273281 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:28.273354 (+ 73us) client_negotiation.cc:769] Sending connection context
0929 18:36:28.273472 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 18:36:28.273608 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2164,"mutex_wait_us":36,"thread_start_us":160,"threads_started":1}
I20260929 18:36:28.274513 15638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:28.267018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41946 (local address 127.28.207.67:41317)
0929 18:36:28.267476 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:28.267491 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:28.268469 (+ 978us) server_negotiation.cc:407] Connection header received
0929 18:36:28.268557 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:28.268562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:28.268746 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:28.268895 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:28.269153 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:28.270255 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:28.271899 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:28.272786 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:28.272885 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:28.273029 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:28.274232 (+ 1203us) server_negotiation.cc:299] Negotiation successful
0929 18:36:28.274345 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":234,"threads_started":1}
I20260929 18:36:28.274783 15637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:28.266509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40552 (local address 127.28.207.65:37863)
0929 18:36:28.266886 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:28.266895 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:28.267514 (+ 619us) server_negotiation.cc:407] Connection header received
0929 18:36:28.267604 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:28.267610 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:28.267833 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:28.267996 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:28.268319 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:28.269498 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:28.271911 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:28.272786 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:28.272874 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:28.273029 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:28.274486 (+ 1457us) server_negotiation.cc:299] Negotiation successful
0929 18:36:28.274636 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":118,"threads_started":1}
I20260929 18:36:28.275249 15635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:28.265861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:37863 (local address 127.0.0.1:40552)
0929 18:36:28.267149 (+ 1288us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:28.267221 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:28.267370 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:28.267989 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:28.267995 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:28.268019 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:28.268174 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:28.268183 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:28.269642 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:28.269646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:28.271731 (+ 2085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:28.271741 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:28.273163 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:28.273167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:28.273294 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:28.273354 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 18:36:28.274998 (+ 1644us) client_negotiation.cc:240] Negotiation successful
0929 18:36:28.275092 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1116,"mutex_wait_us":91,"thread_start_us":112,"threads_started":1}
I20260929 18:36:28.275532 15633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:28.265861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:45153 (local address 127.0.0.1:52842)
0929 18:36:28.267579 (+ 1718us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:28.267592 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:28.267736 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:28.268245 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:28.268248 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:28.268267 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:28.268411 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:28.268421 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:28.270596 (+ 2175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:28.270602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:28.272546 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:28.272560 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:28.274529 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:28.274532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:28.274799 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:28.274839 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 18:36:28.274963 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 18:36:28.275197 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1642,"mutex_wait_us":20,"thread_start_us":120,"threads_started":1}
I20260929 18:36:28.275664 15636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:28.266142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52842 (local address 127.28.207.66:45153)
0929 18:36:28.266979 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:28.266991 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:28.267796 (+ 805us) server_negotiation.cc:407] Connection header received
0929 18:36:28.267908 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:28.267912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:28.268095 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:28.268272 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:28.268515 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:28.269692 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:28.272724 (+ 3032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:28.273887 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:28.274229 (+ 342us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:28.274448 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:28.275365 (+ 917us) server_negotiation.cc:299] Negotiation successful
0929 18:36:28.275503 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":736,"thread_start_us":111,"threads_started":1}
I20260929 18:36:28.278505 15288 tablet_service.cc:1467] Processing CreateTablet for tablet 19f2e43409ad4cc78305fe9016bd6e70 (DEFAULT_TABLE table=test-table [id=30954904de464d74b437ef211314051b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:28.278710 15556 tablet_service.cc:1467] Processing CreateTablet for tablet 19f2e43409ad4cc78305fe9016bd6e70 (DEFAULT_TABLE table=test-table [id=30954904de464d74b437ef211314051b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:28.278710 15422 tablet_service.cc:1467] Processing CreateTablet for tablet 19f2e43409ad4cc78305fe9016bd6e70 (DEFAULT_TABLE table=test-table [id=30954904de464d74b437ef211314051b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:28.279670 15288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f2e43409ad4cc78305fe9016bd6e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:28.279992 15556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f2e43409ad4cc78305fe9016bd6e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:28.279992 15422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19f2e43409ad4cc78305fe9016bd6e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:28.286746 15639 tablet_bootstrap.cc:492] T 19f2e43409ad4cc78305fe9016bd6e70 P b31ca3b0cf384a3192857eac456336bd: Bootstrap starting.
I20260929 18:36:28.289281 15640 tablet_bootstrap.cc:492] T 19f2e43409ad4cc78305fe9016bd6e70 P cc0a416950e64b5da7d7e3fa586d0e4b: Bootstrap starting.
I20260929 18:36:28.290210 15641 tablet_bootstrap.cc:492] T 19f2e43409ad4cc78305fe9016bd6e70 P 4b60271c86ec4a1da15b22c25aaa21b6: Bootstrap starting.
I20260929 18:36:28.290297 15639 tablet_bootstrap.cc:654] T 19f2e43409ad4cc78305fe9016bd6e70 P b31ca3b0cf384a3192857eac456336bd: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:28.291229 15639 log.cc:824] T 19f2e43409ad4cc78305fe9016bd6e70 P b31ca3b0cf384a3192857eac456336bd: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:28.291904 15640 tablet_bootstrap.cc:654] T 19f2e43409ad4cc78305fe9016bd6e70 P cc0a416950e64b5da7d7e3fa586d0e4b: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:28.292572 15641 tablet_bootstrap.cc:654] T 19f2e43409ad4cc78305fe9016bd6e70 P 4b60271c86ec4a1da15b22c25aaa21b6: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:28.293282 15640 log.cc:824] T 19f2e43409ad4cc78305fe9016bd6e70 P cc0a416950e64b5da7d7e3fa586d0e4b: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:28.293856 15639 tablet_bootstrap.cc:492] T 19f2e43409ad4cc78305fe9016bd6e70 P b31ca3b0cf384a3192857eac456336bd: No bootstrap required, opened a new log
I20260929 18:36:28.293872 15641 log.cc:824] T 19f2e43409ad4cc78305fe9016bd6e70 P 4b60271c86ec4a1da15b22c25aaa21b6: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:28.294078 15639 ts_tablet_manager.cc:1397] T 19f2e43409ad4cc78305fe9016bd6e70 P b31ca3b0cf384a3192857eac456336bd: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260929 18:36:28.295543 15640 tablet_bootstrap.cc:492] T 19f2e43409ad4cc78305fe9016bd6e70 P cc0a416950e64b5da7d7e3fa586d0e4b: No bootstrap required, opened a new log
I20260929 18:36:28.295787 15640 ts_tablet_manager.cc:1397] T 19f2e43409ad4cc78305fe9016bd6e70 P cc0a416950e64b5da7d7e3fa586d0e4b: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 18:36:28.296119 15641 tablet_bootstrap.cc:492] T 19f2e43409ad4cc78305fe9016bd6e70 P 4b60271c86ec4a1da15b22c25aaa21b6: No bootstrap required, opened a new log
I20260929 18:36:28.296337 15641 ts_tablet_manager.cc:1397] T 19f2e43409ad4cc78305fe9016bd6e70 P 4b60271c86ec4a1da15b22c25aaa21b6: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 18:36:28.298758 15639 raft_consensus.cc:348] T 19f2e43409ad4cc78305fe9016bd6e70 P b31ca3b0cf384a3192857eac456336bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b60271c86ec4a1da15b22c25aaa21b6" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 45153 } } peers { permanent_uuid: "b31ca3b0cf384a3192857eac456336bd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 37863 } } peers { permanent_uuid: "cc0a416950e64b5da7d7e3fa586d0e4b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 41317 } }
I20260929 18:36:28.299113 15639 raft_consensus.cc:374] T 19f2e43409ad4cc78305fe9016bd6e70 P b31ca3b0cf384a3192857eac456336bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:28.299170 15639 raft_consensus.cc:729] T 19f2e43409ad4cc78305fe9016bd6e70 P b31ca3b0cf384a3192857eac456336bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b31ca3b0cf384a3192857eac456336bd, State: Initialized, Role: FOLLOWER
I20260929 18:36:28.299721 15639 consensus_queue.cc:260] T 19f2e43409ad4cc78305fe9016bd6e70 P b31ca3b0cf384a3192857eac456336bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b60271c86ec4a1da15b22c25aaa21b6" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 45153 } } peers { permanent_uuid: "b31ca3b0cf384a3192857eac456336bd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 37863 } } peers { permanent_uuid: "cc0a416950e64b5da7d7e3fa586d0e4b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 41317 } }
I20260929 18:36:28.300223 15641 raft_consensus.cc:348] T 19f2e43409ad4cc78305fe9016bd6e70 P 4b60271c86ec4a1da15b22c25aaa21b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b60271c86ec4a1da15b22c25aaa21b6" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 45153 } } peers { permanent_uuid: "b31ca3b0cf384a3192857eac456336bd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 37863 } } peers { permanent_uuid: "cc0a416950e64b5da7d7e3fa586d0e4b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 41317 } }
I20260929 18:36:28.300518 15641 raft_consensus.cc:374] T 19f2e43409ad4cc78305fe9016bd6e70 P 4b60271c86ec4a1da15b22c25aaa21b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:28.300563 15641 raft_consensus.cc:729] T 19f2e43409ad4cc78305fe9016bd6e70 P 4b60271c86ec4a1da15b22c25aaa21b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b60271c86ec4a1da15b22c25aaa21b6, State: Initialized, Role: FOLLOWER
I20260929 18:36:28.300685 15354 heartbeater.cc:503] Master 127.28.207.126:40107 was elected leader, sending a full tablet report...
I20260929 18:36:28.300930 15639 ts_tablet_manager.cc:1428] T 19f2e43409ad4cc78305fe9016bd6e70 P b31ca3b0cf384a3192857eac456336bd: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
I20260929 18:36:28.300992 15641 consensus_queue.cc:260] T 19f2e43409ad4cc78305fe9016bd6e70 P 4b60271c86ec4a1da15b22c25aaa21b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b60271c86ec4a1da15b22c25aaa21b6" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 45153 } } peers { permanent_uuid: "b31ca3b0cf384a3192857eac456336bd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 37863 } } peers { permanent_uuid: "cc0a416950e64b5da7d7e3fa586d0e4b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 41317 } }
I20260929 18:36:28.300930 15640 raft_consensus.cc:348] T 19f2e43409ad4cc78305fe9016bd6e70 P cc0a416950e64b5da7d7e3fa586d0e4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b60271c86ec4a1da15b22c25aaa21b6" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 45153 } } peers { permanent_uuid: "b31ca3b0cf384a3192857eac456336bd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 37863 } } peers { permanent_uuid: "cc0a416950e64b5da7d7e3fa586d0e4b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 41317 } }
I20260929 18:36:28.301241 15640 raft_consensus.cc:374] T 19f2e43409ad4cc78305fe9016bd6e70 P cc0a416950e64b5da7d7e3fa586d0e4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:28.301455 15640 raft_consensus.cc:729] T 19f2e43409ad4cc78305fe9016bd6e70 P cc0a416950e64b5da7d7e3fa586d0e4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc0a416950e64b5da7d7e3fa586d0e4b, State: Initialized, Role: FOLLOWER
I20260929 18:36:28.301936 15641 ts_tablet_manager.cc:1428] T 19f2e43409ad4cc78305fe9016bd6e70 P 4b60271c86ec4a1da15b22c25aaa21b6: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 18:36:28.302004 15488 heartbeater.cc:503] Master 127.28.207.126:40107 was elected leader, sending a full tablet report...
I20260929 18:36:28.302062 15640 consensus_queue.cc:260] T 19f2e43409ad4cc78305fe9016bd6e70 P cc0a416950e64b5da7d7e3fa586d0e4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b60271c86ec4a1da15b22c25aaa21b6" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 45153 } } peers { permanent_uuid: "b31ca3b0cf384a3192857eac456336bd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 37863 } } peers { permanent_uuid: "cc0a416950e64b5da7d7e3fa586d0e4b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 41317 } }
I20260929 18:36:28.302891 15622 heartbeater.cc:503] Master 127.28.207.126:40107 was elected leader, sending a full tablet report...
I20260929 18:36:28.304224 15640 ts_tablet_manager.cc:1428] T 19f2e43409ad4cc78305fe9016bd6e70 P cc0a416950e64b5da7d7e3fa586d0e4b: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260929 18:36:28.413661 15647 raft_consensus.cc:482] T 19f2e43409ad4cc78305fe9016bd6e70 P cc0a416950e64b5da7d7e3fa586d0e4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:36:28.413904 15647 raft_consensus.cc:504] T 19f2e43409ad4cc78305fe9016bd6e70 P cc0a416950e64b5da7d7e3fa586d0e4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b60271c86ec4a1da15b22c25aaa21b6" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 45153 } } peers { permanent_uuid: "b31ca3b0cf384a3192857eac456336bd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 37863 } } peers { permanent_uuid: "cc0a416950e64b5da7d7e3fa586d0e4b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 41317 } }
I20260929 18:36:28.415201 15647 leader_election.cc:288] T 19f2e43409ad4cc78305fe9016bd6e70 P cc0a416950e64b5da7d7e3fa586d0e4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b60271c86ec4a1da15b22c25aaa21b6 (127.28.207.66:45153), b31ca3b0cf384a3192857eac456336bd (127.28.207.65:37863)
I20260929 18:36:28.420377 15637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:28.415885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:55067 (local address 127.28.207.65:37863)
0929 18:36:28.416028 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:28.416036 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:28.416057 (+ 21us) server_negotiation.cc:407] Connection header received
0929 18:36:28.416274 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:28.416281 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:28.416356 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:28.416459 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:28.416869 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:28.417506 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:28.419192 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:28.419705 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:28.419814 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:28.419889 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:28.420129 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 18:36:28.420228 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:36:28.420423 15636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:28.415769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:55933 (local address 127.28.207.66:45153)
0929 18:36:28.415954 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:28.415960 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:28.416401 (+ 441us) server_negotiation.cc:407] Connection header received
0929 18:36:28.416514 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:28.416517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:28.416582 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:28.416702 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:28.417220 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:28.417899 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:28.419310 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:28.419786 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:28.419933 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:28.419996 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:28.420213 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 18:36:28.420301 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 18:36:28.420423 15624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:28.415667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:37863 (local address 127.28.207.67:55067)
0929 18:36:28.415972 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:28.415991 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:28.416118 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:28.416507 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:28.416511 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:28.416536 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:28.416723 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:28.416730 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:28.417638 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:28.417641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:28.418994 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:28.419005 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:28.419852 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:28.419857 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:28.419974 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:28.420001 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 18:36:28.420094 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 18:36:28.420214 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187}
I20260929 18:36:28.420456 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:28.415667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:45153 (local address 127.28.207.67:55933)
0929 18:36:28.416215 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:28.416237 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:28.416397 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:28.416760 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:28.416780 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:28.416840 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:28.417027 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:28.417035 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:28.418075 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:28.418078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:28.419173 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:28.419185 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:28.419953 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:28.419956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:28.420034 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:28.420085 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 18:36:28.420183 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 18:36:28.420266 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":32,"thread_start_us":181,"threads_started":1}
I20260929 18:36:28.421438 15308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f2e43409ad4cc78305fe9016bd6e70" candidate_uuid: "cc0a416950e64b5da7d7e3fa586d0e4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b31ca3b0cf384a3192857eac456336bd" is_pre_election: true
I20260929 18:36:28.421895 15308 raft_consensus.cc:2454] T 19f2e43409ad4cc78305fe9016bd6e70 P b31ca3b0cf384a3192857eac456336bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc0a416950e64b5da7d7e3fa586d0e4b in term 0.
I20260929 18:36:28.421907 15442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f2e43409ad4cc78305fe9016bd6e70" candidate_uuid: "cc0a416950e64b5da7d7e3fa586d0e4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b60271c86ec4a1da15b22c25aaa21b6" is_pre_election: true
I20260929 18:36:28.422295 15442 raft_consensus.cc:2454] T 19f2e43409ad4cc78305fe9016bd6e70 P 4b60271c86ec4a1da15b22c25aaa21b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc0a416950e64b5da7d7e3fa586d0e4b in term 0.
I20260929 18:36:28.422528 15511 leader_election.cc:302] T 19f2e43409ad4cc78305fe9016bd6e70 P cc0a416950e64b5da7d7e3fa586d0e4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b31ca3b0cf384a3192857eac456336bd, cc0a416950e64b5da7d7e3fa586d0e4b; no voters:
I20260929 18:36:28.422911 15647 raft_consensus.cc:2781] T 19f2e43409ad4cc78305fe9016bd6e70 P cc0a416950e64b5da7d7e3fa586d0e4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:36:28.423015 15647 raft_consensus.cc:482] T 19f2e43409ad4cc78305fe9016bd6e70 P cc0a416950e64b5da7d7e3fa586d0e4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:36:28.423059 15647 raft_consensus.cc:3037] T 19f2e43409ad4cc78305fe9016bd6e70 P cc0a416950e64b5da7d7e3fa586d0e4b [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:28.424351 15647 raft_consensus.cc:504] T 19f2e43409ad4cc78305fe9016bd6e70 P cc0a416950e64b5da7d7e3fa586d0e4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b60271c86ec4a1da15b22c25aaa21b6" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 45153 } } peers { permanent_uuid: "b31ca3b0cf384a3192857eac456336bd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 37863 } } peers { permanent_uuid: "cc0a416950e64b5da7d7e3fa586d0e4b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 41317 } }
I20260929 18:36:28.424775 15647 leader_election.cc:288] T 19f2e43409ad4cc78305fe9016bd6e70 P cc0a416950e64b5da7d7e3fa586d0e4b [CANDIDATE]: Term 1 election: Requested vote from peers 4b60271c86ec4a1da15b22c25aaa21b6 (127.28.207.66:45153), b31ca3b0cf384a3192857eac456336bd (127.28.207.65:37863)
I20260929 18:36:28.425141 15442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f2e43409ad4cc78305fe9016bd6e70" candidate_uuid: "cc0a416950e64b5da7d7e3fa586d0e4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b60271c86ec4a1da15b22c25aaa21b6"
I20260929 18:36:28.425222 15308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19f2e43409ad4cc78305fe9016bd6e70" candidate_uuid: "cc0a416950e64b5da7d7e3fa586d0e4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b31ca3b0cf384a3192857eac456336bd"
I20260929 18:36:28.425304 15442 raft_consensus.cc:3037] T 19f2e43409ad4cc78305fe9016bd6e70 P 4b60271c86ec4a1da15b22c25aaa21b6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:28.425410 15308 raft_consensus.cc:3037] T 19f2e43409ad4cc78305fe9016bd6e70 P b31ca3b0cf384a3192857eac456336bd [term 0 FOLLOWER]: Advancing to term 1
W20260929 18:36:28.426165 15355 tablet.cc:2369] T 19f2e43409ad4cc78305fe9016bd6e70 P b31ca3b0cf384a3192857eac456336bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:36:28.426704 15442 raft_consensus.cc:2454] T 19f2e43409ad4cc78305fe9016bd6e70 P 4b60271c86ec4a1da15b22c25aaa21b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc0a416950e64b5da7d7e3fa586d0e4b in term 1.
I20260929 18:36:28.427026 15308 raft_consensus.cc:2454] T 19f2e43409ad4cc78305fe9016bd6e70 P b31ca3b0cf384a3192857eac456336bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc0a416950e64b5da7d7e3fa586d0e4b in term 1.
I20260929 18:36:28.427130 15510 leader_election.cc:302] T 19f2e43409ad4cc78305fe9016bd6e70 P cc0a416950e64b5da7d7e3fa586d0e4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b60271c86ec4a1da15b22c25aaa21b6, cc0a416950e64b5da7d7e3fa586d0e4b; no voters:
I20260929 18:36:28.427363 15647 raft_consensus.cc:2781] T 19f2e43409ad4cc78305fe9016bd6e70 P cc0a416950e64b5da7d7e3fa586d0e4b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:28.427670 15647 raft_consensus.cc:686] T 19f2e43409ad4cc78305fe9016bd6e70 P cc0a416950e64b5da7d7e3fa586d0e4b [term 1 LEADER]: Becoming Leader. State: Replica: cc0a416950e64b5da7d7e3fa586d0e4b, State: Running, Role: LEADER
I20260929 18:36:28.428110 15647 consensus_queue.cc:237] T 19f2e43409ad4cc78305fe9016bd6e70 P cc0a416950e64b5da7d7e3fa586d0e4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b60271c86ec4a1da15b22c25aaa21b6" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 45153 } } peers { permanent_uuid: "b31ca3b0cf384a3192857eac456336bd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 37863 } } peers { permanent_uuid: "cc0a416950e64b5da7d7e3fa586d0e4b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 41317 } }
I20260929 18:36:28.431594 15162 catalog_manager.cc:5495] T 19f2e43409ad4cc78305fe9016bd6e70 P cc0a416950e64b5da7d7e3fa586d0e4b reported cstate change: term changed from 0 to 1, leader changed from <none> to cc0a416950e64b5da7d7e3fa586d0e4b (127.28.207.67). New cstate: current_term: 1 leader_uuid: "cc0a416950e64b5da7d7e3fa586d0e4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b60271c86ec4a1da15b22c25aaa21b6" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 45153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b31ca3b0cf384a3192857eac456336bd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 37863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc0a416950e64b5da7d7e3fa586d0e4b" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 41317 } health_report { overall_health: HEALTHY } } }
W20260929 18:36:28.457638 15489 tablet.cc:2369] T 19f2e43409ad4cc78305fe9016bd6e70 P 4b60271c86ec4a1da15b22c25aaa21b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 18:36:28.461333 15623 tablet.cc:2369] T 19f2e43409ad4cc78305fe9016bd6e70 P cc0a416950e64b5da7d7e3fa586d0e4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:36:28.468940 15638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:28.465225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41954 (local address 127.28.207.67:41317)
0929 18:36:28.465395 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:28.465400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:28.465417 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:36:28.465556 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:28.465560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:28.465620 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:28.465734 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:36:28.465995 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:28.466509 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:28.467407 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:28.467622 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:28.467732 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:28.468068 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:36:28.468464 (+ 396us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:36:28.468622 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:28.468728 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 18:36:28.468770 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 18:36:28.476763 15308 raft_consensus.cc:1261] T 19f2e43409ad4cc78305fe9016bd6e70 P b31ca3b0cf384a3192857eac456336bd [term 1 FOLLOWER]: Refusing update from remote peer cc0a416950e64b5da7d7e3fa586d0e4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:36:28.476893 15442 raft_consensus.cc:1261] T 19f2e43409ad4cc78305fe9016bd6e70 P 4b60271c86ec4a1da15b22c25aaa21b6 [term 1 FOLLOWER]: Refusing update from remote peer cc0a416950e64b5da7d7e3fa586d0e4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:36:28.477942 15649 consensus_queue.cc:1035] T 19f2e43409ad4cc78305fe9016bd6e70 P cc0a416950e64b5da7d7e3fa586d0e4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b31ca3b0cf384a3192857eac456336bd" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 37863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:36:28.478214 15647 consensus_queue.cc:1035] T 19f2e43409ad4cc78305fe9016bd6e70 P cc0a416950e64b5da7d7e3fa586d0e4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b60271c86ec4a1da15b22c25aaa21b6" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 45153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:36:28.488245 15653 mvcc.cc:205] Tried to move back new op lower bound from 7334735824791769088 to 7334735824604364800. Current Snapshot: MvccSnapshot[applied={T|T < 7334735824791769088}]
I20260929 18:36:28.489100 15654 mvcc.cc:205] Tried to move back new op lower bound from 7334735824791769088 to 7334735824604364800. Current Snapshot: MvccSnapshot[applied={T|T < 7334735824791769088}]
I20260929 18:36:28.497819 15162 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:36:28.498029 15162 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:36:28.500834 15162 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 424a6f3dc6f1432d8c37b9b493243ea7: Sending DeleteTablet for 3 replicas of tablet 19f2e43409ad4cc78305fe9016bd6e70
I20260929 18:36:28.501827 15556 tablet_service.cc:1514] Processing DeleteTablet for tablet 19f2e43409ad4cc78305fe9016bd6e70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:28 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:41946
I20260929 18:36:28.501832 15288 tablet_service.cc:1514] Processing DeleteTablet for tablet 19f2e43409ad4cc78305fe9016bd6e70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:28 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:40552
I20260929 18:36:28.502408 15422 tablet_service.cc:1514] Processing DeleteTablet for tablet 19f2e43409ad4cc78305fe9016bd6e70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:28 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:52842
I20260929 18:36:28.502586 15663 tablet_replica.cc:330] T 19f2e43409ad4cc78305fe9016bd6e70 P cc0a416950e64b5da7d7e3fa586d0e4b: stopping tablet replica
I20260929 18:36:28.502970 15663 raft_consensus.cc:2229] T 19f2e43409ad4cc78305fe9016bd6e70 P cc0a416950e64b5da7d7e3fa586d0e4b [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:36:28.503243 15662 tablet_replica.cc:330] T 19f2e43409ad4cc78305fe9016bd6e70 P b31ca3b0cf384a3192857eac456336bd: stopping tablet replica
I20260929 18:36:28.503399 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 15225
I20260929 18:36:28.503436 15663 raft_consensus.cc:2258] T 19f2e43409ad4cc78305fe9016bd6e70 P cc0a416950e64b5da7d7e3fa586d0e4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:36:28.507440 15663 ts_tablet_manager.cc:1905] T 19f2e43409ad4cc78305fe9016bd6e70 P cc0a416950e64b5da7d7e3fa586d0e4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:36:28.511201 15664 tablet_replica.cc:330] T 19f2e43409ad4cc78305fe9016bd6e70 P 4b60271c86ec4a1da15b22c25aaa21b6: stopping tablet replica
I20260929 18:36:28.511466 15663 ts_tablet_manager.cc:1918] T 19f2e43409ad4cc78305fe9016bd6e70 P cc0a416950e64b5da7d7e3fa586d0e4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:36:28.511610 15663 log.cc:1196] T 19f2e43409ad4cc78305fe9016bd6e70 P cc0a416950e64b5da7d7e3fa586d0e4b: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790706718260899-29501-0/minicluster-data/ts-2/wal/wals/19f2e43409ad4cc78305fe9016bd6e70
I20260929 18:36:28.511910 15664 raft_consensus.cc:2229] T 19f2e43409ad4cc78305fe9016bd6e70 P 4b60271c86ec4a1da15b22c25aaa21b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:36:28.511979 15663 ts_tablet_manager.cc:1939] T 19f2e43409ad4cc78305fe9016bd6e70 P cc0a416950e64b5da7d7e3fa586d0e4b: Deleting consensus metadata
I20260929 18:36:28.512609 15664 raft_consensus.cc:2258] T 19f2e43409ad4cc78305fe9016bd6e70 P 4b60271c86ec4a1da15b22c25aaa21b6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 18:36:28.512753 15149 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.207.65:37863 (error 108)
I20260929 18:36:28.513410 15147 catalog_manager.cc:4841] TS cc0a416950e64b5da7d7e3fa586d0e4b (127.28.207.67:41317): tablet 19f2e43409ad4cc78305fe9016bd6e70 (table test-table [id=30954904de464d74b437ef211314051b]) successfully deleted
I20260929 18:36:28.513753 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 15359
I20260929 18:36:28.515743 15633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:28.515138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:37863 (local address 127.0.0.1:40564)
0929 18:36:28.515532 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:28.515647 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.65:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305}
W20260929 18:36:28.516011 15149 catalog_manager.cc:4584] TS b31ca3b0cf384a3192857eac456336bd (127.28.207.65:37863): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 19f2e43409ad4cc78305fe9016bd6e70: Network error: Client connection negotiation failed: client connection to 127.28.207.65:37863: connect: Connection refused (error 111)
I20260929 18:36:28.524840 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 15493
I20260929 18:36:28.525178 15633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:28.524764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:45153 (local address 127.0.0.1:52858)
0929 18:36:28.524972 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:28.525045 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.66:45153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 18:36:28.532071 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 15131
2026-09-29T18:36:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2583 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T18:36:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:36:28Z Disabled control of system clock
I20260929 18:36:28.567981 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:32855
--webserver_interface=127.28.207.126
--webserver_port=0
--builtin_ntp_servers=127.28.207.84:37627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:32855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:36:28.718683 15669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:28.719265 15669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:28.719317 15669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:28.724547 15669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:36:28.724653 15669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:28.724669 15669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:28.724689 15669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:36:28.724706 15669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:36:28.730643 15669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:37627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:32855
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:32855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:28.732412 15669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:28.734838 15669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:28.746270 15675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:28.746492 15669 server_base.cc:1034] running on GCE node
W20260929 18:36:28.746325 15674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:28.746774 15677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:28.747605 15669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:28.749337 15669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:28.750609 15669 hybrid_clock.cc:648] HybridClock initialized: now 1790706988750577 us; error 62 us; skew 500 ppm
I20260929 18:36:28.753151 15669 webserver.cc:460] Webserver started at http://127.28.207.126:36795/ using document root <none> and password file <none>
I20260929 18:36:28.753912 15669 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:28.754031 15669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:28.754309 15669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:28.757045 15669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "bb82a83e270c4d11918adfe3fe19ae0f"
format_stamp: "Formatted at 2026-09-29 18:36:28 on dist-test-slave-qcpv"
server_key: "68c368ec3e68ffded9e77746f17866f1"
server_key_iv: "c3fe48d5f73e53239c3fb264c0f45254"
server_key_version: "encryptionkey@0"
I20260929 18:36:28.757726 15669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "bb82a83e270c4d11918adfe3fe19ae0f"
format_stamp: "Formatted at 2026-09-29 18:36:28 on dist-test-slave-qcpv"
server_key: "68c368ec3e68ffded9e77746f17866f1"
server_key_iv: "c3fe48d5f73e53239c3fb264c0f45254"
server_key_version: "encryptionkey@0"
I20260929 18:36:28.762487 15669 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260929 18:36:28.765857 15683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:28.767869 15669 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:36:28.768086 15669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "bb82a83e270c4d11918adfe3fe19ae0f"
format_stamp: "Formatted at 2026-09-29 18:36:28 on dist-test-slave-qcpv"
server_key: "68c368ec3e68ffded9e77746f17866f1"
server_key_iv: "c3fe48d5f73e53239c3fb264c0f45254"
server_key_version: "encryptionkey@0"
I20260929 18:36:28.768239 15669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:28.786724 15669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:28.787763 15669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:28.788005 15669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:28.798312 15669 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:32855
I20260929 18:36:28.798604 15735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:32855 every 8 connection(s)
I20260929 18:36:28.799819 15669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:36:28.803576 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:28.807546 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 15669
I20260929 18:36:28.807693 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:36:28.808053 29501 external_mini_cluster.cc:1428] Setting key 42e942c61442d5f4f3cd5d6cdb524cdb
I20260929 18:36:28.810024 15736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb82a83e270c4d11918adfe3fe19ae0f: Bootstrap starting.
I20260929 18:36:28.812403 15736 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb82a83e270c4d11918adfe3fe19ae0f: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:28.813186 15736 log.cc:824] T 00000000000000000000000000000000 P bb82a83e270c4d11918adfe3fe19ae0f: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:28.815279 15736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb82a83e270c4d11918adfe3fe19ae0f: No bootstrap required, opened a new log
I20260929 18:36:28.819242 15736 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb82a83e270c4d11918adfe3fe19ae0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb82a83e270c4d11918adfe3fe19ae0f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 32855 } }
I20260929 18:36:28.819567 15736 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb82a83e270c4d11918adfe3fe19ae0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:28.819672 15736 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb82a83e270c4d11918adfe3fe19ae0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb82a83e270c4d11918adfe3fe19ae0f, State: Initialized, Role: FOLLOWER
I20260929 18:36:28.820164 15736 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb82a83e270c4d11918adfe3fe19ae0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb82a83e270c4d11918adfe3fe19ae0f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 32855 } }
I20260929 18:36:28.820328 15736 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb82a83e270c4d11918adfe3fe19ae0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:36:28.820420 15736 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb82a83e270c4d11918adfe3fe19ae0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:36:28.820535 15736 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb82a83e270c4d11918adfe3fe19ae0f [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:28.821825 15736 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb82a83e270c4d11918adfe3fe19ae0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb82a83e270c4d11918adfe3fe19ae0f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 32855 } }
I20260929 18:36:28.822268 15736 leader_election.cc:302] T 00000000000000000000000000000000 P bb82a83e270c4d11918adfe3fe19ae0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb82a83e270c4d11918adfe3fe19ae0f; no voters:
I20260929 18:36:28.822646 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:28.809961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38536 (local address 127.28.207.126:32855)
0929 18:36:28.811177 (+ 1216us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:28.811201 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:28.811251 (+ 50us) server_negotiation.cc:407] Connection header received
0929 18:36:28.812628 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:28.812665 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:28.813258 (+ 593us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:28.813793 (+ 535us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:28.814418 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:28.816144 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:28.817297 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:28.817888 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:28.818096 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:28.819427 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:28.819432 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:28.819454 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:28.820188 (+ 734us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:28.820270 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:28.820333 (+ 63us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:28.820529 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:28.820910 (+ 381us) server_negotiation.cc:299] Negotiation successful
0929 18:36:28.821325 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":203,"threads_started":1}
I20260929 18:36:28.822647 15736 leader_election.cc:288] T 00000000000000000000000000000000 P bb82a83e270c4d11918adfe3fe19ae0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:36:28.822839 15741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb82a83e270c4d11918adfe3fe19ae0f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:28.823764 15741 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb82a83e270c4d11918adfe3fe19ae0f [term 1 LEADER]: Becoming Leader. State: Replica: bb82a83e270c4d11918adfe3fe19ae0f, State: Running, Role: LEADER
I20260929 18:36:28.824195 15741 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb82a83e270c4d11918adfe3fe19ae0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb82a83e270c4d11918adfe3fe19ae0f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 32855 } }
I20260929 18:36:28.824298 15736 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb82a83e270c4d11918adfe3fe19ae0f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:36:28.826282 15742 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb82a83e270c4d11918adfe3fe19ae0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb82a83e270c4d11918adfe3fe19ae0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb82a83e270c4d11918adfe3fe19ae0f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 32855 } } }
I20260929 18:36:28.826449 15742 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb82a83e270c4d11918adfe3fe19ae0f [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:28.826797 15743 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb82a83e270c4d11918adfe3fe19ae0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb82a83e270c4d11918adfe3fe19ae0f. Latest consensus state: current_term: 1 leader_uuid: "bb82a83e270c4d11918adfe3fe19ae0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb82a83e270c4d11918adfe3fe19ae0f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 32855 } } }
I20260929 18:36:28.827385 15755 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:36:28.826949 15743 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb82a83e270c4d11918adfe3fe19ae0f [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:28.831133 15755 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:36:28.839318 15755 catalog_manager.cc:1347] Generated new cluster ID: cdb51271aedf46ba9bc19fd3e0a6a0f1
I20260929 18:36:28.839423 15755 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:36:28.871250 15755 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:36:28.872728 15755 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:36:28.882930 15755 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb82a83e270c4d11918adfe3fe19ae0f: Generated new TSK 0
I20260929 18:36:28.883906 15755 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:36:28.897825 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.65:0
--local_ip_for_outbound_sockets=127.28.207.65
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:32855
--builtin_ntp_servers=127.28.207.84:37627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:36:29.053290 15760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:29.053666 15760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:29.053714 15760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:29.059201 15760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:29.059290 15760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:29.059355 15760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.65
I20260929 18:36:29.065044 15760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:37627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:32855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.65
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:29.066691 15760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:29.068506 15760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:29.083441 15766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:29.083436 15765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:29.083432 15768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:29.083928 15760 server_base.cc:1034] running on GCE node
I20260929 18:36:29.084656 15760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:29.085469 15760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:29.086695 15760 hybrid_clock.cc:648] HybridClock initialized: now 1790706989086669 us; error 51 us; skew 500 ppm
I20260929 18:36:29.088932 15760 webserver.cc:460] Webserver started at http://127.28.207.65:36955/ using document root <none> and password file <none>
I20260929 18:36:29.089603 15760 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:29.089704 15760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:29.089948 15760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:29.092406 15760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-0/data/instance:
uuid: "c35807e13820476e826144d234637a5b"
format_stamp: "Formatted at 2026-09-29 18:36:29 on dist-test-slave-qcpv"
server_key: "8334f65c6624cf42b19ca94a621522ae"
server_key_iv: "3e11f537b0f1639c85913d47d82de9ee"
server_key_version: "encryptionkey@0"
I20260929 18:36:29.093026 15760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance:
uuid: "c35807e13820476e826144d234637a5b"
format_stamp: "Formatted at 2026-09-29 18:36:29 on dist-test-slave-qcpv"
server_key: "8334f65c6624cf42b19ca94a621522ae"
server_key_iv: "3e11f537b0f1639c85913d47d82de9ee"
server_key_version: "encryptionkey@0"
I20260929 18:36:29.097141 15760 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:36:29.100754 15774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:29.102581 15760 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:36:29.102756 15760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-0/wal
uuid: "c35807e13820476e826144d234637a5b"
format_stamp: "Formatted at 2026-09-29 18:36:29 on dist-test-slave-qcpv"
server_key: "8334f65c6624cf42b19ca94a621522ae"
server_key_iv: "3e11f537b0f1639c85913d47d82de9ee"
server_key_version: "encryptionkey@0"
I20260929 18:36:29.102974 15760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:29.115375 15760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:29.116101 15760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:29.116285 15760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:29.116906 15760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:29.118013 15760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:29.118069 15760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:29.118155 15760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:29.118180 15760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:29.130048 15760 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.65:39211
I20260929 18:36:29.130075 15887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.65:39211 every 8 connection(s)
I20260929 18:36:29.131217 15760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
I20260929 18:36:29.138324 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 15760
I20260929 18:36:29.138533 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance
I20260929 18:36:29.139092 29501 external_mini_cluster.cc:1428] Setting key a91edc764c0ee5689bb68360483f0884
I20260929 18:36:29.141992 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.66:0
--local_ip_for_outbound_sockets=127.28.207.66
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:32855
--builtin_ntp_servers=127.28.207.84:37627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 18:36:29.143546 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:29.134206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:47737 (local address 127.28.207.126:32855)
0929 18:36:29.134438 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:29.134446 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:29.135956 (+ 1510us) server_negotiation.cc:407] Connection header received
0929 18:36:29.137426 (+ 1470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:29.137429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:29.137491 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:29.137597 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:29.138496 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.139229 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.140524 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.140810 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.140921 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:29.142791 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:29.142797 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:29.142805 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:29.142826 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:29.142872 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:29.142895 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:29.142994 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:29.143233 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 18:36:29.143316 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260929 18:36:29.145166 15890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:29.134840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:32855 (local address 127.28.207.65:47737)
0929 18:36:29.135779 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:29.135824 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:29.137202 (+ 1378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:29.137778 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:29.137785 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:29.137869 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:29.138359 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:29.138372 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.139360 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.139368 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:29.140401 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:29.140410 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.141065 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.141074 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:29.141344 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:29.142407 (+ 1063us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:36:29.142409 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:29.142467 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:36:29.142563 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:29.143033 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:29.143043 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:29.143049 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:36:29.143200 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 18:36:29.143739 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":166,"threads_started":1}
I20260929 18:36:29.146834 15888 heartbeater.cc:347] Connected to a master server at 127.28.207.126:32855
I20260929 18:36:29.147186 15888 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:29.147882 15888 heartbeater.cc:511] Master 127.28.207.126:32855 requested a full tablet report, sending...
I20260929 18:36:29.149868 15700 ts_manager.cc:194] Registered new tserver with Master: c35807e13820476e826144d234637a5b (127.28.207.65:39211)
I20260929 18:36:29.151959 15700 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.207.65:47737
W20260929 18:36:29.260664 15891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:29.260949 15891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:29.260988 15891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:29.265316 15891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:29.265388 15891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:29.265453 15891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.66
I20260929 18:36:29.270450 15891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:37627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:32855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.66
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:29.271790 15891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:29.273418 15891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:29.284773 15899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:29.284827 15896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:29.284801 15897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:29.285104 15891 server_base.cc:1034] running on GCE node
I20260929 18:36:29.285844 15891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:29.286595 15891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:29.287817 15891 hybrid_clock.cc:648] HybridClock initialized: now 1790706989287798 us; error 46 us; skew 500 ppm
I20260929 18:36:29.289934 15891 webserver.cc:460] Webserver started at http://127.28.207.66:32833/ using document root <none> and password file <none>
I20260929 18:36:29.290566 15891 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:29.290666 15891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:29.290969 15891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:29.293099 15891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-1/data/instance:
uuid: "045f47b03955419c97e5adf675e2c156"
format_stamp: "Formatted at 2026-09-29 18:36:29 on dist-test-slave-qcpv"
server_key: "4223b7a2009b7915184c802d354b0490"
server_key_iv: "3ee8a5ff8524afd2259f79f3529dd5a9"
server_key_version: "encryptionkey@0"
I20260929 18:36:29.293680 15891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance:
uuid: "045f47b03955419c97e5adf675e2c156"
format_stamp: "Formatted at 2026-09-29 18:36:29 on dist-test-slave-qcpv"
server_key: "4223b7a2009b7915184c802d354b0490"
server_key_iv: "3ee8a5ff8524afd2259f79f3529dd5a9"
server_key_version: "encryptionkey@0"
I20260929 18:36:29.297973 15891 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:36:29.301571 15905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:29.303224 15891 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:36:29.303398 15891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-1/wal
uuid: "045f47b03955419c97e5adf675e2c156"
format_stamp: "Formatted at 2026-09-29 18:36:29 on dist-test-slave-qcpv"
server_key: "4223b7a2009b7915184c802d354b0490"
server_key_iv: "3ee8a5ff8524afd2259f79f3529dd5a9"
server_key_version: "encryptionkey@0"
I20260929 18:36:29.303530 15891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:29.314247 15891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:29.314965 15891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:29.315160 15891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:29.315752 15891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:29.316962 15891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:29.317040 15891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:29.317112 15891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:29.317139 15891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:29.330173 15891 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.66:39365
I20260929 18:36:29.330199 16018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.66:39365 every 8 connection(s)
I20260929 18:36:29.331180 15891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
I20260929 18:36:29.340808 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 15891
I20260929 18:36:29.340929 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance
I20260929 18:36:29.341229 29501 external_mini_cluster.cc:1428] Setting key 68099d882ab1533f3266aa071f612eba
I20260929 18:36:29.342773 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:29.333616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:39193 (local address 127.28.207.126:32855)
0929 18:36:29.333809 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:29.333813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:29.335220 (+ 1407us) server_negotiation.cc:407] Connection header received
0929 18:36:29.337020 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:29.337024 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:29.337096 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:29.337218 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:29.338332 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.339010 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.340315 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.340549 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.340666 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:29.342125 (+ 1459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:29.342129 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:29.342131 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:29.342145 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:29.342173 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:29.342183 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:29.342263 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:29.342487 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 18:36:29.342606 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 18:36:29.343823 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.67:0
--local_ip_for_outbound_sockets=127.28.207.67
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:32855
--builtin_ntp_servers=127.28.207.84:37627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 18:36:29.344429 16021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:29.334023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:32855 (local address 127.28.207.66:39193)
0929 18:36:29.335050 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:29.335096 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:29.336713 (+ 1617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:29.337441 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:29.337450 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:29.337557 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:29.338170 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:29.338184 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.339151 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.339154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:29.340161 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:29.340171 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.340726 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.340731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:29.340970 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:29.341799 (+ 829us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:36:29.341801 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:29.341851 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:36:29.341933 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:29.342291 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:29.342301 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:29.342308 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 18:36:29.342472 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 18:36:29.343310 (+ 838us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":125,"threads_started":1}
I20260929 18:36:29.345925 16019 heartbeater.cc:347] Connected to a master server at 127.28.207.126:32855
I20260929 18:36:29.346212 16019 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:29.346953 16019 heartbeater.cc:511] Master 127.28.207.126:32855 requested a full tablet report, sending...
I20260929 18:36:29.348258 15700 ts_manager.cc:194] Registered new tserver with Master: 045f47b03955419c97e5adf675e2c156 (127.28.207.66:39365)
I20260929 18:36:29.349180 15700 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.207.66:39193
W20260929 18:36:29.488596 16022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:29.488973 16022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:29.489020 16022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:29.493928 16022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:29.494026 16022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:29.494135 16022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.67
I20260929 18:36:29.501137 16022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:37627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:32855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.67
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:29.502998 16022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:29.504877 16022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:29.517745 16027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:29.517828 16030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:29.517876 16028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:29.518431 16022 server_base.cc:1034] running on GCE node
I20260929 18:36:29.519107 16022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:29.519987 16022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:29.521204 16022 hybrid_clock.cc:648] HybridClock initialized: now 1790706989521161 us; error 58 us; skew 500 ppm
I20260929 18:36:29.523927 16022 webserver.cc:460] Webserver started at http://127.28.207.67:41111/ using document root <none> and password file <none>
I20260929 18:36:29.524708 16022 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:29.524781 16022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:29.525051 16022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:29.528168 16022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-2/data/instance:
uuid: "0683177e75784751af24172600c70c47"
format_stamp: "Formatted at 2026-09-29 18:36:29 on dist-test-slave-qcpv"
server_key: "4f1e7a1ed1078a5e06ff861e96383575"
server_key_iv: "b17b51fcfbd04b6d9634347c4fc2ad9d"
server_key_version: "encryptionkey@0"
I20260929 18:36:29.529022 16022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance:
uuid: "0683177e75784751af24172600c70c47"
format_stamp: "Formatted at 2026-09-29 18:36:29 on dist-test-slave-qcpv"
server_key: "4f1e7a1ed1078a5e06ff861e96383575"
server_key_iv: "b17b51fcfbd04b6d9634347c4fc2ad9d"
server_key_version: "encryptionkey@0"
I20260929 18:36:29.534116 16022 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 18:36:29.537685 16036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:29.539896 16022 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260929 18:36:29.540108 16022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-2/wal
uuid: "0683177e75784751af24172600c70c47"
format_stamp: "Formatted at 2026-09-29 18:36:29 on dist-test-slave-qcpv"
server_key: "4f1e7a1ed1078a5e06ff861e96383575"
server_key_iv: "b17b51fcfbd04b6d9634347c4fc2ad9d"
server_key_version: "encryptionkey@0"
I20260929 18:36:29.540395 16022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:29.573149 16022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:29.574448 16022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:29.574720 16022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:29.575562 16022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:29.576785 16022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:29.576866 16022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:29.576941 16022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:29.576992 16022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:29.590127 16022 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.67:42551
I20260929 18:36:29.590153 16149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.67:42551 every 8 connection(s)
I20260929 18:36:29.591331 16022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
I20260929 18:36:29.594811 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 16022
I20260929 18:36:29.595057 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance
I20260929 18:36:29.595379 29501 external_mini_cluster.cc:1428] Setting key 65345034fb2da0742cd5ac34bc121f5f
I20260929 18:36:29.603473 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:29.593681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:39167 (local address 127.28.207.126:32855)
0929 18:36:29.593980 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:29.593985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:29.595380 (+ 1395us) server_negotiation.cc:407] Connection header received
0929 18:36:29.597397 (+ 2017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:29.597400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:29.597454 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:29.597520 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:29.598793 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.599367 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.600854 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.601104 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.601201 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:29.602656 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:29.602659 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:29.602665 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:29.602682 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:29.602703 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:29.602711 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:29.602792 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:29.603297 (+ 505us) server_negotiation.cc:299] Negotiation successful
0929 18:36:29.603351 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260929 18:36:29.605307 16152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:29.594105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:32855 (local address 127.28.207.67:39167)
0929 18:36:29.595134 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:29.595204 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:29.597104 (+ 1900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:29.597741 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:29.597751 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:29.597885 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:29.598629 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:29.598645 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.599527 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.599530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:29.600714 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:29.600725 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.601315 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.601319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:29.601503 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:29.602347 (+ 844us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:36:29.602348 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:29.602399 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:36:29.602483 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:29.602842 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:29.602851 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:29.602995 (+ 144us) client_negotiation.cc:769] Sending connection context
0929 18:36:29.603291 (+ 296us) client_negotiation.cc:240] Negotiation successful
0929 18:36:29.603733 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":149,"threads_started":1}
I20260929 18:36:29.607406 16150 heartbeater.cc:347] Connected to a master server at 127.28.207.126:32855
I20260929 18:36:29.607703 16150 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:29.608579 16150 heartbeater.cc:511] Master 127.28.207.126:32855 requested a full tablet report, sending...
I20260929 18:36:29.609937 15700 ts_manager.cc:194] Registered new tserver with Master: 0683177e75784751af24172600c70c47 (127.28.207.67:42551)
I20260929 18:36:29.611178 29501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:36:29.611338 15700 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.207.67:39167
I20260929 18:36:29.619212 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:29.613959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38552 (local address 127.28.207.126:32855)
0929 18:36:29.614118 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:29.614129 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:29.614214 (+ 85us) server_negotiation.cc:407] Connection header received
0929 18:36:29.614353 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:29.614355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:29.614392 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:29.614462 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:29.615090 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.615736 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.616801 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.617050 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.617268 (+ 218us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:29.618390 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:29.618393 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:29.618395 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:29.618408 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:29.618430 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:29.618438 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:29.618564 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:29.618905 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 18:36:29.619030 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 18:36:29.623569 15700 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38552:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:36:29.627234 15700 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:36:29.645015 16163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:29.638525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47810 (local address 127.28.207.66:39365)
0929 18:36:29.639151 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:29.639166 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:29.639792 (+ 626us) server_negotiation.cc:407] Connection header received
0929 18:36:29.639902 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:29.639909 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:29.640141 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:29.640315 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:29.640805 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.641721 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.643250 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.644072 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.644184 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:29.644397 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:29.644699 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 18:36:29.644822 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":188,"threads_started":1}
I20260929 18:36:29.645071 16162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:29.638278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:39365 (local address 127.0.0.1:47810)
0929 18:36:29.639405 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:29.639466 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:29.639616 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:29.640352 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:29.640359 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:29.640393 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:29.640664 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:29.640672 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.641828 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.641834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:29.643111 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:29.643124 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.644224 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.644228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:29.644446 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:29.644507 (+ 61us) client_negotiation.cc:769] Sending connection context
0929 18:36:29.644647 (+ 140us) client_negotiation.cc:240] Negotiation successful
0929 18:36:29.644838 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":918,"mutex_wait_us":136,"spinlock_wait_cycles":4736,"thread_start_us":130,"threads_started":1}
I20260929 18:36:29.645753 16164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:29.638537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:42551 (local address 127.0.0.1:59200)
0929 18:36:29.639441 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:29.639466 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:29.639616 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:29.640355 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:29.640359 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:29.640395 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:29.640667 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:29.640676 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.642762 (+ 2086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.642766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:29.643933 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:29.643944 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.645327 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.645330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:29.645443 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:29.645473 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 18:36:29.645529 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 18:36:29.645619 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"mutex_wait_us":29,"spinlock_wait_cycles":14208,"thread_start_us":105,"threads_started":1}
I20260929 18:36:29.645999 16161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:29.639145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:39211 (local address 127.0.0.1:38436)
0929 18:36:29.640036 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:29.640053 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:29.640167 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:29.641418 (+ 1251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:29.641421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:29.641442 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:29.641536 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:29.641542 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.643197 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.643199 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:29.644383 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:29.644392 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.645538 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.645542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:29.645643 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:29.645684 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 18:36:29.645736 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 18:36:29.645833 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"mutex_wait_us":4,"thread_start_us":113,"threads_started":1}
I20260929 18:36:29.646833 16166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:29.639952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38436 (local address 127.28.207.65:39211)
0929 18:36:29.640623 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:29.640631 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:29.640657 (+ 26us) server_negotiation.cc:407] Connection header received
0929 18:36:29.640748 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:29.640754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:29.641212 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:29.641419 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:29.641665 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.642917 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.644509 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.645388 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.646342 (+ 954us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:29.646573 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:29.646633 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 18:36:29.646721 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":138,"threads_started":1}
I20260929 18:36:29.646963 16165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:29.639356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59200 (local address 127.28.207.67:42551)
0929 18:36:29.639957 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:29.639969 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:29.639988 (+ 19us) server_negotiation.cc:407] Connection header received
0929 18:36:29.640053 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:29.640058 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:29.640204 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:29.640341 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:29.640822 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.642586 (+ 1764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.644099 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.644973 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.646005 (+ 1032us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:29.646392 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:29.646460 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 18:36:29.646612 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":184,"threads_started":1}
I20260929 18:36:29.650038 15822 tablet_service.cc:1467] Processing CreateTablet for tablet 6b0d0428337b48e39ce68d6b59fd1a64 (DEFAULT_TABLE table=test-table [id=9af2f2002e694adcb00d3812bc45a024]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:29.650038 15953 tablet_service.cc:1467] Processing CreateTablet for tablet 6b0d0428337b48e39ce68d6b59fd1a64 (DEFAULT_TABLE table=test-table [id=9af2f2002e694adcb00d3812bc45a024]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:29.650085 16084 tablet_service.cc:1467] Processing CreateTablet for tablet 6b0d0428337b48e39ce68d6b59fd1a64 (DEFAULT_TABLE table=test-table [id=9af2f2002e694adcb00d3812bc45a024]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:29.651322 15953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b0d0428337b48e39ce68d6b59fd1a64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:29.651322 15822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b0d0428337b48e39ce68d6b59fd1a64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:29.651474 16084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b0d0428337b48e39ce68d6b59fd1a64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:29.659214 16167 tablet_bootstrap.cc:492] T 6b0d0428337b48e39ce68d6b59fd1a64 P 0683177e75784751af24172600c70c47: Bootstrap starting.
I20260929 18:36:29.660388 16169 tablet_bootstrap.cc:492] T 6b0d0428337b48e39ce68d6b59fd1a64 P 045f47b03955419c97e5adf675e2c156: Bootstrap starting.
I20260929 18:36:29.660831 16168 tablet_bootstrap.cc:492] T 6b0d0428337b48e39ce68d6b59fd1a64 P c35807e13820476e826144d234637a5b: Bootstrap starting.
I20260929 18:36:29.662598 16167 tablet_bootstrap.cc:654] T 6b0d0428337b48e39ce68d6b59fd1a64 P 0683177e75784751af24172600c70c47: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:29.662909 16169 tablet_bootstrap.cc:654] T 6b0d0428337b48e39ce68d6b59fd1a64 P 045f47b03955419c97e5adf675e2c156: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:29.662909 16168 tablet_bootstrap.cc:654] T 6b0d0428337b48e39ce68d6b59fd1a64 P c35807e13820476e826144d234637a5b: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:29.663604 16167 log.cc:824] T 6b0d0428337b48e39ce68d6b59fd1a64 P 0683177e75784751af24172600c70c47: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:29.663838 16168 log.cc:824] T 6b0d0428337b48e39ce68d6b59fd1a64 P c35807e13820476e826144d234637a5b: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:29.663838 16169 log.cc:824] T 6b0d0428337b48e39ce68d6b59fd1a64 P 045f47b03955419c97e5adf675e2c156: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:29.666208 16169 tablet_bootstrap.cc:492] T 6b0d0428337b48e39ce68d6b59fd1a64 P 045f47b03955419c97e5adf675e2c156: No bootstrap required, opened a new log
I20260929 18:36:29.666209 16167 tablet_bootstrap.cc:492] T 6b0d0428337b48e39ce68d6b59fd1a64 P 0683177e75784751af24172600c70c47: No bootstrap required, opened a new log
I20260929 18:36:29.666352 16168 tablet_bootstrap.cc:492] T 6b0d0428337b48e39ce68d6b59fd1a64 P c35807e13820476e826144d234637a5b: No bootstrap required, opened a new log
I20260929 18:36:29.666429 16167 ts_tablet_manager.cc:1397] T 6b0d0428337b48e39ce68d6b59fd1a64 P 0683177e75784751af24172600c70c47: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260929 18:36:29.666429 16169 ts_tablet_manager.cc:1397] T 6b0d0428337b48e39ce68d6b59fd1a64 P 045f47b03955419c97e5adf675e2c156: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260929 18:36:29.666616 16168 ts_tablet_manager.cc:1397] T 6b0d0428337b48e39ce68d6b59fd1a64 P c35807e13820476e826144d234637a5b: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260929 18:36:29.671113 16168 raft_consensus.cc:348] T 6b0d0428337b48e39ce68d6b59fd1a64 P c35807e13820476e826144d234637a5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f47b03955419c97e5adf675e2c156" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 39365 } } peers { permanent_uuid: "0683177e75784751af24172600c70c47" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 42551 } } peers { permanent_uuid: "c35807e13820476e826144d234637a5b" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 39211 } }
I20260929 18:36:29.671103 16167 raft_consensus.cc:348] T 6b0d0428337b48e39ce68d6b59fd1a64 P 0683177e75784751af24172600c70c47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f47b03955419c97e5adf675e2c156" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 39365 } } peers { permanent_uuid: "0683177e75784751af24172600c70c47" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 42551 } } peers { permanent_uuid: "c35807e13820476e826144d234637a5b" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 39211 } }
I20260929 18:36:29.671103 16169 raft_consensus.cc:348] T 6b0d0428337b48e39ce68d6b59fd1a64 P 045f47b03955419c97e5adf675e2c156 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f47b03955419c97e5adf675e2c156" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 39365 } } peers { permanent_uuid: "0683177e75784751af24172600c70c47" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 42551 } } peers { permanent_uuid: "c35807e13820476e826144d234637a5b" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 39211 } }
I20260929 18:36:29.671422 16169 raft_consensus.cc:374] T 6b0d0428337b48e39ce68d6b59fd1a64 P 045f47b03955419c97e5adf675e2c156 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:29.671422 16167 raft_consensus.cc:374] T 6b0d0428337b48e39ce68d6b59fd1a64 P 0683177e75784751af24172600c70c47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:29.671447 16168 raft_consensus.cc:374] T 6b0d0428337b48e39ce68d6b59fd1a64 P c35807e13820476e826144d234637a5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:29.671487 16167 raft_consensus.cc:729] T 6b0d0428337b48e39ce68d6b59fd1a64 P 0683177e75784751af24172600c70c47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0683177e75784751af24172600c70c47, State: Initialized, Role: FOLLOWER
I20260929 18:36:29.671487 16169 raft_consensus.cc:729] T 6b0d0428337b48e39ce68d6b59fd1a64 P 045f47b03955419c97e5adf675e2c156 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 045f47b03955419c97e5adf675e2c156, State: Initialized, Role: FOLLOWER
I20260929 18:36:29.671509 16168 raft_consensus.cc:729] T 6b0d0428337b48e39ce68d6b59fd1a64 P c35807e13820476e826144d234637a5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c35807e13820476e826144d234637a5b, State: Initialized, Role: FOLLOWER
I20260929 18:36:29.672066 16168 consensus_queue.cc:260] T 6b0d0428337b48e39ce68d6b59fd1a64 P c35807e13820476e826144d234637a5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f47b03955419c97e5adf675e2c156" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 39365 } } peers { permanent_uuid: "0683177e75784751af24172600c70c47" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 42551 } } peers { permanent_uuid: "c35807e13820476e826144d234637a5b" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 39211 } }
I20260929 18:36:29.672071 16167 consensus_queue.cc:260] T 6b0d0428337b48e39ce68d6b59fd1a64 P 0683177e75784751af24172600c70c47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f47b03955419c97e5adf675e2c156" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 39365 } } peers { permanent_uuid: "0683177e75784751af24172600c70c47" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 42551 } } peers { permanent_uuid: "c35807e13820476e826144d234637a5b" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 39211 } }
I20260929 18:36:29.672071 16169 consensus_queue.cc:260] T 6b0d0428337b48e39ce68d6b59fd1a64 P 045f47b03955419c97e5adf675e2c156 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f47b03955419c97e5adf675e2c156" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 39365 } } peers { permanent_uuid: "0683177e75784751af24172600c70c47" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 42551 } } peers { permanent_uuid: "c35807e13820476e826144d234637a5b" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 39211 } }
I20260929 18:36:29.673273 16167 ts_tablet_manager.cc:1428] T 6b0d0428337b48e39ce68d6b59fd1a64 P 0683177e75784751af24172600c70c47: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260929 18:36:29.673278 16019 heartbeater.cc:503] Master 127.28.207.126:32855 was elected leader, sending a full tablet report...
I20260929 18:36:29.673480 16168 ts_tablet_manager.cc:1428] T 6b0d0428337b48e39ce68d6b59fd1a64 P c35807e13820476e826144d234637a5b: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260929 18:36:29.673727 16150 heartbeater.cc:503] Master 127.28.207.126:32855 was elected leader, sending a full tablet report...
I20260929 18:36:29.673265 16169 ts_tablet_manager.cc:1428] T 6b0d0428337b48e39ce68d6b59fd1a64 P 045f47b03955419c97e5adf675e2c156: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260929 18:36:29.674803 15888 heartbeater.cc:503] Master 127.28.207.126:32855 was elected leader, sending a full tablet report...
W20260929 18:36:29.832661 16020 tablet.cc:2369] T 6b0d0428337b48e39ce68d6b59fd1a64 P 045f47b03955419c97e5adf675e2c156: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:36:29.834189 16175 raft_consensus.cc:482] T 6b0d0428337b48e39ce68d6b59fd1a64 P c35807e13820476e826144d234637a5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:36:29.834451 16175 raft_consensus.cc:504] T 6b0d0428337b48e39ce68d6b59fd1a64 P c35807e13820476e826144d234637a5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f47b03955419c97e5adf675e2c156" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 39365 } } peers { permanent_uuid: "0683177e75784751af24172600c70c47" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 42551 } } peers { permanent_uuid: "c35807e13820476e826144d234637a5b" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 39211 } }
I20260929 18:36:29.835845 16175 leader_election.cc:288] T 6b0d0428337b48e39ce68d6b59fd1a64 P c35807e13820476e826144d234637a5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 045f47b03955419c97e5adf675e2c156 (127.28.207.66:39365), 0683177e75784751af24172600c70c47 (127.28.207.67:42551)
I20260929 18:36:29.840639 16163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:29.836284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:37783 (local address 127.28.207.66:39365)
0929 18:36:29.836477 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:29.836484 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:29.836759 (+ 275us) server_negotiation.cc:407] Connection header received
0929 18:36:29.836956 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:29.836959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:29.837019 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:29.837114 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:29.837443 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.838030 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.839516 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.839977 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.840131 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:29.840190 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:29.840400 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 18:36:29.840483 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 18:36:29.840669 16176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:29.836110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:39365 (local address 127.28.207.65:37783)
0929 18:36:29.836625 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:29.836650 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:29.836820 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:29.837134 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:29.837137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:29.837162 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:29.837316 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:29.837323 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.838163 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.838166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:29.839366 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:29.839379 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.840126 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.840130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:29.840242 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:29.840270 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 18:36:29.840381 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 18:36:29.840483 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":1,"thread_start_us":175,"threads_started":1}
I20260929 18:36:29.840981 16165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:29.836508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:41543 (local address 127.28.207.67:42551)
0929 18:36:29.836722 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:29.836727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:29.837055 (+ 328us) server_negotiation.cc:407] Connection header received
0929 18:36:29.837257 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:29.837260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:29.837315 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:29.837408 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:29.837716 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.838268 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.839745 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.840202 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.840369 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:29.840434 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:29.840620 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 18:36:29.840701 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260929 18:36:29.841413 16177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:29.836388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:42551 (local address 127.28.207.65:41543)
0929 18:36:29.836935 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:29.836949 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:29.837127 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:29.837421 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:29.837424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:29.837442 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:29.837600 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:29.837606 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.838394 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.838399 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:29.839593 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:29.839604 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.840369 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.840373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:29.840490 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:29.840510 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 18:36:29.840601 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 18:36:29.840690 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":35,"thread_start_us":174,"threads_started":1}
I20260929 18:36:29.841683 15967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0d0428337b48e39ce68d6b59fd1a64" candidate_uuid: "c35807e13820476e826144d234637a5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "045f47b03955419c97e5adf675e2c156" is_pre_election: true
I20260929 18:36:29.842186 15967 raft_consensus.cc:2454] T 6b0d0428337b48e39ce68d6b59fd1a64 P 045f47b03955419c97e5adf675e2c156 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c35807e13820476e826144d234637a5b in term 0.
I20260929 18:36:29.842325 16104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0d0428337b48e39ce68d6b59fd1a64" candidate_uuid: "c35807e13820476e826144d234637a5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0683177e75784751af24172600c70c47" is_pre_election: true
W20260929 18:36:29.842566 16151 tablet.cc:2369] T 6b0d0428337b48e39ce68d6b59fd1a64 P 0683177e75784751af24172600c70c47: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:36:29.842783 16104 raft_consensus.cc:2454] T 6b0d0428337b48e39ce68d6b59fd1a64 P 0683177e75784751af24172600c70c47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c35807e13820476e826144d234637a5b in term 0.
I20260929 18:36:29.842818 15775 leader_election.cc:302] T 6b0d0428337b48e39ce68d6b59fd1a64 P c35807e13820476e826144d234637a5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 045f47b03955419c97e5adf675e2c156, c35807e13820476e826144d234637a5b; no voters:
I20260929 18:36:29.843187 16175 raft_consensus.cc:2781] T 6b0d0428337b48e39ce68d6b59fd1a64 P c35807e13820476e826144d234637a5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:36:29.843333 16175 raft_consensus.cc:482] T 6b0d0428337b48e39ce68d6b59fd1a64 P c35807e13820476e826144d234637a5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:36:29.843411 16175 raft_consensus.cc:3037] T 6b0d0428337b48e39ce68d6b59fd1a64 P c35807e13820476e826144d234637a5b [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:29.844813 16175 raft_consensus.cc:504] T 6b0d0428337b48e39ce68d6b59fd1a64 P c35807e13820476e826144d234637a5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f47b03955419c97e5adf675e2c156" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 39365 } } peers { permanent_uuid: "0683177e75784751af24172600c70c47" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 42551 } } peers { permanent_uuid: "c35807e13820476e826144d234637a5b" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 39211 } }
I20260929 18:36:29.845389 16175 leader_election.cc:288] T 6b0d0428337b48e39ce68d6b59fd1a64 P c35807e13820476e826144d234637a5b [CANDIDATE]: Term 1 election: Requested vote from peers 045f47b03955419c97e5adf675e2c156 (127.28.207.66:39365), 0683177e75784751af24172600c70c47 (127.28.207.67:42551)
I20260929 18:36:29.845994 15967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0d0428337b48e39ce68d6b59fd1a64" candidate_uuid: "c35807e13820476e826144d234637a5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "045f47b03955419c97e5adf675e2c156"
I20260929 18:36:29.846148 15967 raft_consensus.cc:3037] T 6b0d0428337b48e39ce68d6b59fd1a64 P 045f47b03955419c97e5adf675e2c156 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:29.847537 15967 raft_consensus.cc:2454] T 6b0d0428337b48e39ce68d6b59fd1a64 P 045f47b03955419c97e5adf675e2c156 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c35807e13820476e826144d234637a5b in term 1.
I20260929 18:36:29.847972 16104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b0d0428337b48e39ce68d6b59fd1a64" candidate_uuid: "c35807e13820476e826144d234637a5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0683177e75784751af24172600c70c47"
I20260929 18:36:29.848088 15775 leader_election.cc:302] T 6b0d0428337b48e39ce68d6b59fd1a64 P c35807e13820476e826144d234637a5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 045f47b03955419c97e5adf675e2c156, c35807e13820476e826144d234637a5b; no voters:
I20260929 18:36:29.848148 16104 raft_consensus.cc:3037] T 6b0d0428337b48e39ce68d6b59fd1a64 P 0683177e75784751af24172600c70c47 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:29.848428 16175 raft_consensus.cc:2781] T 6b0d0428337b48e39ce68d6b59fd1a64 P c35807e13820476e826144d234637a5b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:29.848747 16175 raft_consensus.cc:686] T 6b0d0428337b48e39ce68d6b59fd1a64 P c35807e13820476e826144d234637a5b [term 1 LEADER]: Becoming Leader. State: Replica: c35807e13820476e826144d234637a5b, State: Running, Role: LEADER
I20260929 18:36:29.849417 16175 consensus_queue.cc:237] T 6b0d0428337b48e39ce68d6b59fd1a64 P c35807e13820476e826144d234637a5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f47b03955419c97e5adf675e2c156" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 39365 } } peers { permanent_uuid: "0683177e75784751af24172600c70c47" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 42551 } } peers { permanent_uuid: "c35807e13820476e826144d234637a5b" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 39211 } }
I20260929 18:36:29.849578 16104 raft_consensus.cc:2454] T 6b0d0428337b48e39ce68d6b59fd1a64 P 0683177e75784751af24172600c70c47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c35807e13820476e826144d234637a5b in term 1.
I20260929 18:36:29.854702 15700 catalog_manager.cc:5495] T 6b0d0428337b48e39ce68d6b59fd1a64 P c35807e13820476e826144d234637a5b reported cstate change: term changed from 0 to 1, leader changed from <none> to c35807e13820476e826144d234637a5b (127.28.207.65). New cstate: current_term: 1 leader_uuid: "c35807e13820476e826144d234637a5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "045f47b03955419c97e5adf675e2c156" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 39365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0683177e75784751af24172600c70c47" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 42551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c35807e13820476e826144d234637a5b" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 39211 } health_report { overall_health: HEALTHY } } }
W20260929 18:36:29.883109 15889 tablet.cc:2369] T 6b0d0428337b48e39ce68d6b59fd1a64 P c35807e13820476e826144d234637a5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:36:29.890825 16166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:29.886145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38446 (local address 127.28.207.65:39211)
0929 18:36:29.886353 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:29.886357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:29.886376 (+ 19us) server_negotiation.cc:407] Connection header received
0929 18:36:29.886537 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:29.886540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:29.886616 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:29.886743 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:36:29.887131 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.887815 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.888807 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:29.889056 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:29.889157 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:29.889488 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:36:29.890133 (+ 645us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:36:29.890270 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:29.890542 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 18:36:29.890604 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 18:36:29.899199 16104 raft_consensus.cc:1261] T 6b0d0428337b48e39ce68d6b59fd1a64 P 0683177e75784751af24172600c70c47 [term 1 FOLLOWER]: Refusing update from remote peer c35807e13820476e826144d234637a5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:36:29.899204 15967 raft_consensus.cc:1261] T 6b0d0428337b48e39ce68d6b59fd1a64 P 045f47b03955419c97e5adf675e2c156 [term 1 FOLLOWER]: Refusing update from remote peer c35807e13820476e826144d234637a5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:36:29.900229 16178 consensus_queue.cc:1035] T 6b0d0428337b48e39ce68d6b59fd1a64 P c35807e13820476e826144d234637a5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0683177e75784751af24172600c70c47" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 42551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:36:29.900488 16175 consensus_queue.cc:1035] T 6b0d0428337b48e39ce68d6b59fd1a64 P c35807e13820476e826144d234637a5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "045f47b03955419c97e5adf675e2c156" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 39365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:36:29.919166 15700 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:36:29.919428 15700 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:36:29.923113 15700 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb82a83e270c4d11918adfe3fe19ae0f: Sending DeleteTablet for 3 replicas of tablet 6b0d0428337b48e39ce68d6b59fd1a64
I20260929 18:36:29.924407 15822 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b0d0428337b48e39ce68d6b59fd1a64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:29 UTC) from {username='slave'} at 127.0.0.1:38436
I20260929 18:36:29.925133 16190 tablet_replica.cc:330] T 6b0d0428337b48e39ce68d6b59fd1a64 P c35807e13820476e826144d234637a5b: stopping tablet replica
I20260929 18:36:29.925819 16190 raft_consensus.cc:2229] T 6b0d0428337b48e39ce68d6b59fd1a64 P c35807e13820476e826144d234637a5b [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:36:29.926522 16190 raft_consensus.cc:2258] T 6b0d0428337b48e39ce68d6b59fd1a64 P c35807e13820476e826144d234637a5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:36:29.926559 15953 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b0d0428337b48e39ce68d6b59fd1a64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:29 UTC) from {username='slave'} at 127.0.0.1:47810
I20260929 18:36:29.927026 16191 tablet_replica.cc:330] T 6b0d0428337b48e39ce68d6b59fd1a64 P 045f47b03955419c97e5adf675e2c156: stopping tablet replica
I20260929 18:36:29.927392 16191 raft_consensus.cc:2229] T 6b0d0428337b48e39ce68d6b59fd1a64 P 045f47b03955419c97e5adf675e2c156 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:36:29.927449 16084 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b0d0428337b48e39ce68d6b59fd1a64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:29 UTC) from {username='slave'} at 127.0.0.1:59200
I20260929 18:36:29.927881 16191 raft_consensus.cc:2258] T 6b0d0428337b48e39ce68d6b59fd1a64 P 045f47b03955419c97e5adf675e2c156 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:36:29.929179 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 15760
I20260929 18:36:29.930147 16191 ts_tablet_manager.cc:1905] T 6b0d0428337b48e39ce68d6b59fd1a64 P 045f47b03955419c97e5adf675e2c156: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:36:29.933311 16191 ts_tablet_manager.cc:1918] T 6b0d0428337b48e39ce68d6b59fd1a64 P 045f47b03955419c97e5adf675e2c156: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:36:29.933403 16191 log.cc:1196] T 6b0d0428337b48e39ce68d6b59fd1a64 P 045f47b03955419c97e5adf675e2c156: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-1/wal/wals/6b0d0428337b48e39ce68d6b59fd1a64
I20260929 18:36:29.933722 16191 ts_tablet_manager.cc:1939] T 6b0d0428337b48e39ce68d6b59fd1a64 P 045f47b03955419c97e5adf675e2c156: Deleting consensus metadata
I20260929 18:36:29.935144 16192 tablet_replica.cc:330] T 6b0d0428337b48e39ce68d6b59fd1a64 P 0683177e75784751af24172600c70c47: stopping tablet replica
I20260929 18:36:29.935166 15684 catalog_manager.cc:4841] TS 045f47b03955419c97e5adf675e2c156 (127.28.207.66:39365): tablet 6b0d0428337b48e39ce68d6b59fd1a64 (table test-table [id=9af2f2002e694adcb00d3812bc45a024]) successfully deleted
I20260929 18:36:29.935525 16192 raft_consensus.cc:2229] T 6b0d0428337b48e39ce68d6b59fd1a64 P 0683177e75784751af24172600c70c47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:36:29.935873 16192 raft_consensus.cc:2258] T 6b0d0428337b48e39ce68d6b59fd1a64 P 0683177e75784751af24172600c70c47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:36:29.937251 16192 ts_tablet_manager.cc:1905] T 6b0d0428337b48e39ce68d6b59fd1a64 P 0683177e75784751af24172600c70c47: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 18:36:29.939647 15684 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.207.65:39211 (error 108)
I20260929 18:36:29.939759 16192 ts_tablet_manager.cc:1918] T 6b0d0428337b48e39ce68d6b59fd1a64 P 0683177e75784751af24172600c70c47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:36:29.939874 16192 log.cc:1196] T 6b0d0428337b48e39ce68d6b59fd1a64 P 0683177e75784751af24172600c70c47: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790706718260899-29501-0/minicluster-data/ts-2/wal/wals/6b0d0428337b48e39ce68d6b59fd1a64
I20260929 18:36:29.940191 16192 ts_tablet_manager.cc:1939] T 6b0d0428337b48e39ce68d6b59fd1a64 P 0683177e75784751af24172600c70c47: Deleting consensus metadata
I20260929 18:36:29.940486 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 15891
I20260929 18:36:29.942150 15685 catalog_manager.cc:4841] TS 0683177e75784751af24172600c70c47 (127.28.207.67:42551): tablet 6b0d0428337b48e39ce68d6b59fd1a64 (table test-table [id=9af2f2002e694adcb00d3812bc45a024]) successfully deleted
I20260929 18:36:29.943445 16161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:29.942904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:39211 (local address 127.0.0.1:38454)
0929 18:36:29.943209 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:29.943317 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.65:39211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
W20260929 18:36:29.943790 15684 catalog_manager.cc:4584] TS c35807e13820476e826144d234637a5b (127.28.207.65:39211): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6b0d0428337b48e39ce68d6b59fd1a64: Network error: Client connection negotiation failed: client connection to 127.28.207.65:39211: connect: Connection refused (error 111)
I20260929 18:36:29.949097 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 16022
I20260929 18:36:29.955575 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 15669
2026-09-29T18:36:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1424 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:36:30 dist-test-slave-qcpv krb5kdc[16196](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:36:30 dist-test-slave-qcpv krb5kdc[16196](info): set up 2 sockets
Sep 29 18:36:30 dist-test-slave-qcpv krb5kdc[16196](info): commencing operation
krb5kdc: starting...
W20260929 18:36:31.031687 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:36:31 dist-test-slave-qcpv krb5kdc[16196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:36:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:36:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.207.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
I20260929 18:36:31.178251 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:33309
--webserver_interface=127.28.207.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:46265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:33309
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:31.316370 16209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:31.316748 16209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:31.316862 16209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:31.321184 16209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:36:31.321297 16209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:31.321338 16209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:31.321399 16209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:36:31.321465 16209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:36:31.327523 16209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:46265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:33309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:33309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:31.328792 16209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:31.329923 16209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:31.342423 16217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:31.342442 16215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:31.342442 16214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:31.342764 16209 server_base.cc:1034] running on GCE node
I20260929 18:36:31.343626 16209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:31.345204 16209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:31.346462 16209 hybrid_clock.cc:648] HybridClock initialized: now 1790706991346434 us; error 54 us; skew 500 ppm
Sep 29 18:36:31 dist-test-slave-qcpv krb5kdc[16196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706991, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:31.350593 16209 init.cc:377] Logged in from keytab as kudu/127.28.207.126@KRBTEST.COM (short username kudu)
I20260929 18:36:31.352576 16209 webserver.cc:460] Webserver started at http://127.28.207.126:46621/ using document root <none> and password file <none>
I20260929 18:36:31.353390 16209 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:31.353513 16209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:31.353806 16209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:31.356880 16209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "a1322574edd44567ba375a4c1dab5d8e"
format_stamp: "Formatted at 2026-09-29 18:36:31 on dist-test-slave-qcpv"
server_key: "9492b89aba72062baaceab939ecb5f3d"
server_key_iv: "a845f2ed51a3922aa657b179d19e38a8"
server_key_version: "encryptionkey@0"
I20260929 18:36:31.357707 16209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "a1322574edd44567ba375a4c1dab5d8e"
format_stamp: "Formatted at 2026-09-29 18:36:31 on dist-test-slave-qcpv"
server_key: "9492b89aba72062baaceab939ecb5f3d"
server_key_iv: "a845f2ed51a3922aa657b179d19e38a8"
server_key_version: "encryptionkey@0"
I20260929 18:36:31.363077 16209 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 18:36:31.366537 16224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:31.368413 16209 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:36:31.368635 16209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "a1322574edd44567ba375a4c1dab5d8e"
format_stamp: "Formatted at 2026-09-29 18:36:31 on dist-test-slave-qcpv"
server_key: "9492b89aba72062baaceab939ecb5f3d"
server_key_iv: "a845f2ed51a3922aa657b179d19e38a8"
server_key_version: "encryptionkey@0"
I20260929 18:36:31.368806 16209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:31.395623 16209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:31.400184 16209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:31.400440 16209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:31.410516 16209 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:33309
I20260929 18:36:31.410545 16276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:33309 every 8 connection(s)
I20260929 18:36:31.411847 16209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:36:31.414992 16277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:31.417371 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 16209
I20260929 18:36:31.417555 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:36:31.417912 29501 external_mini_cluster.cc:1428] Setting key beb892b090582c0180e481b9b4e17517
I20260929 18:36:31.421500 16277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1322574edd44567ba375a4c1dab5d8e: Bootstrap starting.
I20260929 18:36:31.423895 16277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1322574edd44567ba375a4c1dab5d8e: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:31.424693 16277 log.cc:824] T 00000000000000000000000000000000 P a1322574edd44567ba375a4c1dab5d8e: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:31.426651 16277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1322574edd44567ba375a4c1dab5d8e: No bootstrap required, opened a new log
Sep 29 18:36:31 dist-test-slave-qcpv krb5kdc[16196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:36:31.430933 16277 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1322574edd44567ba375a4c1dab5d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1322574edd44567ba375a4c1dab5d8e" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33309 } }
I20260929 18:36:31.431224 16277 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1322574edd44567ba375a4c1dab5d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:31.431279 16277 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1322574edd44567ba375a4c1dab5d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1322574edd44567ba375a4c1dab5d8e, State: Initialized, Role: FOLLOWER
I20260929 18:36:31.431794 16277 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1322574edd44567ba375a4c1dab5d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1322574edd44567ba375a4c1dab5d8e" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33309 } }
I20260929 18:36:31.431990 16277 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1322574edd44567ba375a4c1dab5d8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:36:31.432060 16277 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1322574edd44567ba375a4c1dab5d8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:36:31.432188 16277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1322574edd44567ba375a4c1dab5d8e [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:31.433145 16277 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1322574edd44567ba375a4c1dab5d8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1322574edd44567ba375a4c1dab5d8e" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33309 } }
I20260929 18:36:31.433493 16277 leader_election.cc:302] T 00000000000000000000000000000000 P a1322574edd44567ba375a4c1dab5d8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1322574edd44567ba375a4c1dab5d8e; no voters:
I20260929 18:36:31.433826 16277 leader_election.cc:288] T 00000000000000000000000000000000 P a1322574edd44567ba375a4c1dab5d8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:36:31.434044 16282 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1322574edd44567ba375a4c1dab5d8e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:31.434382 16282 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1322574edd44567ba375a4c1dab5d8e [term 1 LEADER]: Becoming Leader. State: Replica: a1322574edd44567ba375a4c1dab5d8e, State: Running, Role: LEADER
I20260929 18:36:31.435000 16282 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1322574edd44567ba375a4c1dab5d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1322574edd44567ba375a4c1dab5d8e" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33309 } }
I20260929 18:36:31.435395 16277 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1322574edd44567ba375a4c1dab5d8e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:36:31.437654 16283 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1322574edd44567ba375a4c1dab5d8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1322574edd44567ba375a4c1dab5d8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1322574edd44567ba375a4c1dab5d8e" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33309 } } }
I20260929 18:36:31.437819 16283 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1322574edd44567ba375a4c1dab5d8e [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:31.438118 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:31.420017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38728 (local address 127.28.207.126:33309)
0929 18:36:31.420609 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:31.420631 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:31.420682 (+ 51us) server_negotiation.cc:407] Connection header received
0929 18:36:31.421814 (+ 1132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:31.421841 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:31.422323 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:31.422771 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:31.423827 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:31.425235 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:31.426275 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:31.427010 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:31.427224 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:31.430703 (+ 3479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:31.430711 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:31.430732 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:31.430774 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:31.433422 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:31.434091 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:31.434102 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:31.434112 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:31.434303 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:31.434743 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:31.434747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:31.434748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:31.435339 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:31.435580 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:31.436171 (+ 591us) server_negotiation.cc:299] Negotiation successful
0929 18:36:31.436584 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":161,"threads_started":1}
I20260929 18:36:31.438426 16284 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1322574edd44567ba375a4c1dab5d8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1322574edd44567ba375a4c1dab5d8e. Latest consensus state: current_term: 1 leader_uuid: "a1322574edd44567ba375a4c1dab5d8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1322574edd44567ba375a4c1dab5d8e" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33309 } } }
I20260929 18:36:31.439360 16284 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1322574edd44567ba375a4c1dab5d8e [sys.catalog]: This master's current role is: LEADER
W20260929 18:36:31.440899 16296 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a1322574edd44567ba375a4c1dab5d8e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:36:31.440999 16296 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:36:31.441125 16297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:36:31.441924 16297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:36:31.449400 16297 catalog_manager.cc:1347] Generated new cluster ID: 5b163bb70d4a42a5af5ff4e20b1a60c5
I20260929 18:36:31.449503 16297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:36:31.466873 16297 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:36:31.468124 16297 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:36:31.473853 16297 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1322574edd44567ba375a4c1dab5d8e: Generated new TSK 0
I20260929 18:36:31.474679 16297 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.28.207.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab.
Entry for principal kudu/127.28.207.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab.
I20260929 18:36:31.519030 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.65:0
--local_ip_for_outbound_sockets=127.28.207.65
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:33309
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:46265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:31.642724 16303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:31.643151 16303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:31.643203 16303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:31.647565 16303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:31.647646 16303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:31.647710 16303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.65
I20260929 18:36:31.653337 16303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:46265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.65.keytab
--principal=kudu/127.28.207.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:33309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.65
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:31.654408 16303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:31.655542 16303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:31.666254 16308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:31.666254 16309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:31.666301 16311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:31.666604 16303 server_base.cc:1034] running on GCE node
I20260929 18:36:31.667124 16303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:31.667804 16303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:31.669124 16303 hybrid_clock.cc:648] HybridClock initialized: now 1790706991669064 us; error 82 us; skew 500 ppm
Sep 29 18:36:31 dist-test-slave-qcpv krb5kdc[16196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706991, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:31.672425 16303 init.cc:377] Logged in from keytab as kudu/127.28.207.65@KRBTEST.COM (short username kudu)
I20260929 18:36:31.673545 16303 webserver.cc:460] Webserver started at http://127.28.207.65:41147/ using document root <none> and password file <none>
I20260929 18:36:31.674140 16303 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:31.674212 16303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:31.674388 16303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:31.676606 16303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-0/data/instance:
uuid: "35271bfc58544ae6b541e8d1d99dd3af"
format_stamp: "Formatted at 2026-09-29 18:36:31 on dist-test-slave-qcpv"
server_key: "8980107b4938d0dac1455746023955dc"
server_key_iv: "287e54705785813d37277eb773fdd6a3"
server_key_version: "encryptionkey@0"
I20260929 18:36:31.677130 16303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance:
uuid: "35271bfc58544ae6b541e8d1d99dd3af"
format_stamp: "Formatted at 2026-09-29 18:36:31 on dist-test-slave-qcpv"
server_key: "8980107b4938d0dac1455746023955dc"
server_key_iv: "287e54705785813d37277eb773fdd6a3"
server_key_version: "encryptionkey@0"
I20260929 18:36:31.681224 16303 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 18:36:31.684077 16318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:31.685606 16303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 18:36:31.685729 16303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-0/wal
uuid: "35271bfc58544ae6b541e8d1d99dd3af"
format_stamp: "Formatted at 2026-09-29 18:36:31 on dist-test-slave-qcpv"
server_key: "8980107b4938d0dac1455746023955dc"
server_key_iv: "287e54705785813d37277eb773fdd6a3"
server_key_version: "encryptionkey@0"
I20260929 18:36:31.685889 16303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:31.714674 16303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:31.718974 16303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:31.719167 16303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:31.719933 16303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:31.720888 16303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:31.720968 16303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:31.721014 16303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:31.721029 16303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:31.733906 16303 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.65:36945
I20260929 18:36:31.734045 16431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.65:36945 every 8 connection(s)
I20260929 18:36:31.735229 16303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
I20260929 18:36:31.737555 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 16303
I20260929 18:36:31.737730 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance
I20260929 18:36:31.738134 29501 external_mini_cluster.cc:1428] Setting key a3aa3a516312faf0eb6f7d6c28137ff6
Sep 29 18:36:31 dist-test-slave-qcpv krb5kdc[16196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706991, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.65@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.207.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.66@KRBTEST.COM" created.
I20260929 18:36:31.758143 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:31.738234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:37909 (local address 127.28.207.126:33309)
0929 18:36:31.738414 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:31.738417 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:31.740969 (+ 2552us) server_negotiation.cc:407] Connection header received
0929 18:36:31.742995 (+ 2026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:31.743006 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:31.743109 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:31.743272 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:31.744724 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:31.745770 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:31.747504 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:31.748191 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:31.748407 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:31.752671 (+ 4264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:31.752690 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:31.752699 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:31.752762 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:31.754684 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:31.755533 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:31.755536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:31.755538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:31.755641 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:31.756148 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:31.756152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:31.756156 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:31.756518 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:31.756667 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:31.757648 (+ 981us) server_negotiation.cc:299] Negotiation successful
0929 18:36:31.757969 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 18:36:31.759110 16434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:31.739204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:33309 (local address 127.28.207.65:37909)
0929 18:36:31.740752 (+ 1548us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:31.740846 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:31.742530 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:31.743485 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:31.743494 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:31.743946 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:31.744509 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:31.744524 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:31.746020 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:31.746023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:31.747310 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:31.747325 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:31.748475 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:31.748478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:31.748668 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:31.749756 (+ 1088us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:31.749776 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:31.752145 (+ 2369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:31.754951 (+ 2806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:31.754958 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:31.754974 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:31.755328 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:31.755745 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:31.755747 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:31.755749 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:31.755942 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:31.756669 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:31.756675 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:31.757088 (+ 413us) client_negotiation.cc:769] Sending connection context
0929 18:36:31.757350 (+ 262us) client_negotiation.cc:240] Negotiation successful
0929 18:36:31.757782 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1172,"thread_start_us":895,"threads_started":1}
I20260929 18:36:31.761080 16432 heartbeater.cc:347] Connected to a master server at 127.28.207.126:33309
I20260929 18:36:31.761475 16432 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:31.762338 16432 heartbeater.cc:511] Master 127.28.207.126:33309 requested a full tablet report, sending...
I20260929 18:36:31.764586 16241 ts_manager.cc:194] Registered new tserver with Master: 35271bfc58544ae6b541e8d1d99dd3af (127.28.207.65:36945)
I20260929 18:36:31.765915 16241 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.65@KRBTEST.COM'} at 127.28.207.65:37909
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab.
Entry for principal kudu/127.28.207.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab.
I20260929 18:36:31.779165 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.66:0
--local_ip_for_outbound_sockets=127.28.207.66
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:33309
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:46265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:31.923522 16437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:31.923965 16437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:31.924055 16437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:31.929159 16437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:31.929296 16437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:31.929406 16437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.66
I20260929 18:36:31.936077 16437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:46265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.66.keytab
--principal=kudu/127.28.207.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:33309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.66
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:31.937731 16437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:31.939078 16437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:31.951205 16442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:31.951202 16445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:31.951239 16443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:31.951647 16437 server_base.cc:1034] running on GCE node
I20260929 18:36:31.952247 16437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:31.952935 16437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:31.954144 16437 hybrid_clock.cc:648] HybridClock initialized: now 1790706991954125 us; error 40 us; skew 500 ppm
Sep 29 18:36:31 dist-test-slave-qcpv krb5kdc[16196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706991, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:31.957506 16437 init.cc:377] Logged in from keytab as kudu/127.28.207.66@KRBTEST.COM (short username kudu)
I20260929 18:36:31.958551 16437 webserver.cc:460] Webserver started at http://127.28.207.66:37321/ using document root <none> and password file <none>
I20260929 18:36:31.959254 16437 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:31.959348 16437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:31.959574 16437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:31.961967 16437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-1/data/instance:
uuid: "c62d9465adfe4a9ca6c13b227a3726ef"
format_stamp: "Formatted at 2026-09-29 18:36:31 on dist-test-slave-qcpv"
server_key: "4c7049c82e31c5ff76346db04b85bacd"
server_key_iv: "a6aaab8ad9685f5c98974d281e681cbf"
server_key_version: "encryptionkey@0"
I20260929 18:36:31.962728 16437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance:
uuid: "c62d9465adfe4a9ca6c13b227a3726ef"
format_stamp: "Formatted at 2026-09-29 18:36:31 on dist-test-slave-qcpv"
server_key: "4c7049c82e31c5ff76346db04b85bacd"
server_key_iv: "a6aaab8ad9685f5c98974d281e681cbf"
server_key_version: "encryptionkey@0"
I20260929 18:36:31.968063 16437 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260929 18:36:31.970995 16452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:31.972610 16437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:36:31.972792 16437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-1/wal
uuid: "c62d9465adfe4a9ca6c13b227a3726ef"
format_stamp: "Formatted at 2026-09-29 18:36:31 on dist-test-slave-qcpv"
server_key: "4c7049c82e31c5ff76346db04b85bacd"
server_key_iv: "a6aaab8ad9685f5c98974d281e681cbf"
server_key_version: "encryptionkey@0"
I20260929 18:36:31.972929 16437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:31.997104 16437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:32.000722 16437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:32.000946 16437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:32.001600 16437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:32.002542 16437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:32.002622 16437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:32.002699 16437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:32.002743 16437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:32.014366 16437 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.66:37137
I20260929 18:36:32.014724 16565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.66:37137 every 8 connection(s)
I20260929 18:36:32.015659 16437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
I20260929 18:36:32.019150 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 16437
I20260929 18:36:32.019328 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance
I20260929 18:36:32.019675 29501 external_mini_cluster.cc:1428] Setting key 665a63e2041befd55c1e479a61af90e7
Sep 29 18:36:32 dist-test-slave-qcpv krb5kdc[16196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706991, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.66@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.207.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.67@KRBTEST.COM" created.
I20260929 18:36:32.037506 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:32.018247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:46517 (local address 127.28.207.126:33309)
0929 18:36:32.018613 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:32.018618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:32.020635 (+ 2017us) server_negotiation.cc:407] Connection header received
0929 18:36:32.022142 (+ 1507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:32.022145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:32.022217 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:32.022352 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:32.024575 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.025642 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.027217 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.027656 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.027800 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:32.031617 (+ 3817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:32.031621 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:32.031624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:32.031665 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:32.033912 (+ 2247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:32.034830 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:32.034834 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:32.034836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:32.034989 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:32.035925 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:32.035932 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:32.035936 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:32.036268 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:32.036399 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:32.037139 (+ 740us) server_negotiation.cc:299] Negotiation successful
0929 18:36:32.037309 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260929 18:36:32.038699 16568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:32.018931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:33309 (local address 127.28.207.66:46517)
0929 18:36:32.020347 (+ 1416us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:32.020447 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:32.021760 (+ 1313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:32.022659 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:32.022674 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:32.023371 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:32.024290 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:32.024321 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.025852 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.025856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:32.027061 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:32.027074 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.027992 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.027996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:32.028203 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:32.029087 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:32.029109 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:32.031344 (+ 2235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:32.034139 (+ 2795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:32.034147 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:32.034165 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:32.034604 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:32.035224 (+ 620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:32.035226 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:32.035228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:32.035450 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:32.036406 (+ 956us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:32.036418 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:32.036831 (+ 413us) client_negotiation.cc:769] Sending connection context
0929 18:36:32.037080 (+ 249us) client_negotiation.cc:240] Negotiation successful
0929 18:36:32.037590 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1097,"thread_start_us":901,"threads_started":1}
I20260929 18:36:32.040393 16566 heartbeater.cc:347] Connected to a master server at 127.28.207.126:33309
I20260929 18:36:32.040717 16566 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:32.041342 16566 heartbeater.cc:511] Master 127.28.207.126:33309 requested a full tablet report, sending...
I20260929 18:36:32.043054 16241 ts_manager.cc:194] Registered new tserver with Master: c62d9465adfe4a9ca6c13b227a3726ef (127.28.207.66:37137)
I20260929 18:36:32.044018 16241 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.66@KRBTEST.COM'} at 127.28.207.66:46517
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab.
Entry for principal kudu/127.28.207.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab.
I20260929 18:36:32.054370 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.67:0
--local_ip_for_outbound_sockets=127.28.207.67
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:33309
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:46265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:32.174446 16571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:32.174726 16571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:32.174768 16571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:32.179131 16571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:32.179198 16571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:32.179257 16571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.67
I20260929 18:36:32.184584 16571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:46265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.67.keytab
--principal=kudu/127.28.207.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:33309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.67
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:32.185591 16571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:32.186565 16571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:32.197160 16576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:32.197139 16579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:32.197168 16577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:32.197463 16571 server_base.cc:1034] running on GCE node
I20260929 18:36:32.198000 16571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:32.198688 16571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:32.199954 16571 hybrid_clock.cc:648] HybridClock initialized: now 1790706992199917 us; error 71 us; skew 500 ppm
Sep 29 18:36:32 dist-test-slave-qcpv krb5kdc[16196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706992, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:32.203254 16571 init.cc:377] Logged in from keytab as kudu/127.28.207.67@KRBTEST.COM (short username kudu)
I20260929 18:36:32.204246 16571 webserver.cc:460] Webserver started at http://127.28.207.67:37695/ using document root <none> and password file <none>
I20260929 18:36:32.204846 16571 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:32.204939 16571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:32.205153 16571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:32.207360 16571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-2/data/instance:
uuid: "4e1b114af8e445659ca8ca73b2dc1817"
format_stamp: "Formatted at 2026-09-29 18:36:32 on dist-test-slave-qcpv"
server_key: "2f86005f3890eb5ad049ee5d076e5b72"
server_key_iv: "b1ba674e5b873777c087e2f9e2c96eda"
server_key_version: "encryptionkey@0"
I20260929 18:36:32.207918 16571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance:
uuid: "4e1b114af8e445659ca8ca73b2dc1817"
format_stamp: "Formatted at 2026-09-29 18:36:32 on dist-test-slave-qcpv"
server_key: "2f86005f3890eb5ad049ee5d076e5b72"
server_key_iv: "b1ba674e5b873777c087e2f9e2c96eda"
server_key_version: "encryptionkey@0"
I20260929 18:36:32.212144 16571 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 18:36:32.215492 16586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:32.217401 16571 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 18:36:32.217597 16571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-2/wal
uuid: "4e1b114af8e445659ca8ca73b2dc1817"
format_stamp: "Formatted at 2026-09-29 18:36:32 on dist-test-slave-qcpv"
server_key: "2f86005f3890eb5ad049ee5d076e5b72"
server_key_iv: "b1ba674e5b873777c087e2f9e2c96eda"
server_key_version: "encryptionkey@0"
I20260929 18:36:32.217736 16571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:32.232561 16571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:32.236634 16571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:32.237042 16571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:32.237754 16571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:32.238714 16571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:32.238761 16571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:32.238802 16571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:32.238818 16571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:32.250433 16571 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.67:43137
I20260929 18:36:32.250582 16699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.67:43137 every 8 connection(s)
I20260929 18:36:32.251466 16571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
I20260929 18:36:32.252610 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 16571
I20260929 18:36:32.252766 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance
I20260929 18:36:32.253065 29501 external_mini_cluster.cc:1428] Setting key 05ac2a7512bac170fa63c4772d447158
Sep 29 18:36:32 dist-test-slave-qcpv krb5kdc[16196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706992, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.67@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:36:32.269515 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:32.254108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:56067 (local address 127.28.207.126:33309)
0929 18:36:32.254269 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:32.254272 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:32.255488 (+ 1216us) server_negotiation.cc:407] Connection header received
0929 18:36:32.257172 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:32.257176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:32.257286 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:32.257493 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:32.258764 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.259579 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.260941 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.261194 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.261301 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:32.264662 (+ 3361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:32.264666 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:32.264668 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:32.264699 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:32.266518 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:32.267355 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:32.267359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:32.267361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:32.267479 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:32.267969 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:32.267971 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:32.267972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:32.268201 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:32.268331 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:32.269173 (+ 842us) server_negotiation.cc:299] Negotiation successful
0929 18:36:32.269328 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:36:32.270695 16702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:32.254667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:33309 (local address 127.28.207.67:56067)
0929 18:36:32.255294 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:32.255354 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:32.256867 (+ 1513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:32.257631 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:32.257638 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:32.258030 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:32.258551 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:32.258564 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.259765 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.259771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:32.260781 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:32.260793 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.261390 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.261393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:32.261564 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:32.262385 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:36:32.262402 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:32.264401 (+ 1999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:32.266796 (+ 2395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:32.266804 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:32.266823 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:32.267226 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:32.267607 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:36:32.267610 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:36:32.267612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:36:32.267845 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:36:32.268359 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:32.268370 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:32.268841 (+ 471us) client_negotiation.cc:769] Sending connection context
0929 18:36:32.269126 (+ 285us) client_negotiation.cc:240] Negotiation successful
0929 18:36:32.269605 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":209,"threads_started":1}
I20260929 18:36:32.272370 16700 heartbeater.cc:347] Connected to a master server at 127.28.207.126:33309
I20260929 18:36:32.272744 16700 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:32.273489 16700 heartbeater.cc:511] Master 127.28.207.126:33309 requested a full tablet report, sending...
I20260929 18:36:32.275146 16241 ts_manager.cc:194] Registered new tserver with Master: 4e1b114af8e445659ca8ca73b2dc1817 (127.28.207.67:43137)
I20260929 18:36:32.275821 16241 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.28.207.67@KRBTEST.COM'} at 127.28.207.67:56067
I20260929 18:36:32.281665 29501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:36:32.293519 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:32.284037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38744 (local address 127.28.207.126:33309)
0929 18:36:32.284230 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:32.284234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:32.284531 (+ 297us) server_negotiation.cc:407] Connection header received
0929 18:36:32.284598 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:32.284601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:32.284666 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:32.284769 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:32.285592 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.286167 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.287213 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.287477 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.287602 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:32.289335 (+ 1733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:32.289339 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:32.289342 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:32.289370 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:32.291340 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:32.291905 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:32.291908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:32.291909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:32.292013 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:32.292433 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:32.292436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:32.292437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:32.292752 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:32.292886 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:32.293222 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 18:36:32.293362 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260929 18:36:32.297912 16241 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:36:32.302290 16241 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:36:32.321192 16715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:32.314141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:37137 (local address 127.0.0.1:33498)
0929 18:36:32.315161 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:32.315179 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:32.315419 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:32.316177 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:32.316180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:32.316197 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:32.316429 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:32.316438 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.317707 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.317713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:32.319277 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:32.319287 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.320498 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.320502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:32.320652 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:32.320703 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 18:36:32.320839 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 18:36:32.320994 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":947,"mutex_wait_us":2,"thread_start_us":103,"threads_started":1}
I20260929 18:36:32.321924 16714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:32.314679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33498 (local address 127.28.207.66:37137)
0929 18:36:32.315757 (+ 1078us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:32.315775 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:32.315794 (+ 19us) server_negotiation.cc:407] Connection header received
0929 18:36:32.315861 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:32.315865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:32.316034 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:32.316186 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:32.316588 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.317575 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.319509 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.320287 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.320415 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:32.320572 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:32.321072 (+ 500us) server_negotiation.cc:299] Negotiation successful
0929 18:36:32.321176 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":978,"thread_start_us":172,"threads_started":1}
I20260929 18:36:32.321979 16712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:32.313808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41442 (local address 127.28.207.67:43137)
0929 18:36:32.314398 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:32.314418 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:32.315076 (+ 658us) server_negotiation.cc:407] Connection header received
0929 18:36:32.315160 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:32.315165 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:32.315395 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:32.315568 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:32.316092 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.317221 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.319900 (+ 2679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.320713 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.320837 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:32.320958 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:32.321707 (+ 749us) server_negotiation.cc:299] Negotiation successful
0929 18:36:32.321813 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":214,"threads_started":1}
I20260929 18:36:32.322192 16711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:32.313564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:43137 (local address 127.0.0.1:41442)
0929 18:36:32.314606 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:32.314672 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:32.314831 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:32.315756 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:32.315763 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:32.315792 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:32.315951 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:32.315959 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.317382 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.317386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:32.319752 (+ 2366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:32.319764 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.320839 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.320843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:32.320955 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:32.320995 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 18:36:32.321364 (+ 369us) client_negotiation.cc:240] Negotiation successful
0929 18:36:32.321478 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":16,"thread_start_us":97,"threads_started":1}
I20260929 18:36:32.322940 16713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:32.314746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:36945 (local address 127.0.0.1:49298)
0929 18:36:32.315818 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:32.315830 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:32.315947 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:32.316621 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:32.316624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:32.316643 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:32.316732 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:32.316739 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.318052 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.318056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:32.319885 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:32.319895 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.322483 (+ 2588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.322490 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:32.322562 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:32.322583 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 18:36:32.322681 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 18:36:32.322771 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1017,"mutex_wait_us":73,"thread_start_us":160,"threads_started":1}
I20260929 18:36:32.323086 16716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:32.314957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49298 (local address 127.28.207.65:36945)
0929 18:36:32.315551 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:32.315566 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:32.315936 (+ 370us) server_negotiation.cc:407] Connection header received
0929 18:36:32.316081 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:32.316086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:32.316435 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:32.316628 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:32.316840 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.317907 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.320011 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.321619 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.321745 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:32.321942 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:32.322725 (+ 783us) server_negotiation.cc:299] Negotiation successful
0929 18:36:32.322888 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":120,"threads_started":1}
I20260929 18:36:32.324908 16496 tablet_service.cc:1467] Processing CreateTablet for tablet b440bcfdc1894f2c8209b85a2f26307f (DEFAULT_TABLE table=test-table [id=c0bddcd6b8f8401fb9d499a742c812a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:32.325354 16634 tablet_service.cc:1467] Processing CreateTablet for tablet b440bcfdc1894f2c8209b85a2f26307f (DEFAULT_TABLE table=test-table [id=c0bddcd6b8f8401fb9d499a742c812a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:32.326045 16496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b440bcfdc1894f2c8209b85a2f26307f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:32.326551 16634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b440bcfdc1894f2c8209b85a2f26307f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:32.329077 16366 tablet_service.cc:1467] Processing CreateTablet for tablet b440bcfdc1894f2c8209b85a2f26307f (DEFAULT_TABLE table=test-table [id=c0bddcd6b8f8401fb9d499a742c812a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:32.330467 16366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b440bcfdc1894f2c8209b85a2f26307f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:32.332757 16717 tablet_bootstrap.cc:492] T b440bcfdc1894f2c8209b85a2f26307f P c62d9465adfe4a9ca6c13b227a3726ef: Bootstrap starting.
I20260929 18:36:32.334986 16717 tablet_bootstrap.cc:654] T b440bcfdc1894f2c8209b85a2f26307f P c62d9465adfe4a9ca6c13b227a3726ef: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:32.337503 16717 log.cc:824] T b440bcfdc1894f2c8209b85a2f26307f P c62d9465adfe4a9ca6c13b227a3726ef: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:32.337627 16718 tablet_bootstrap.cc:492] T b440bcfdc1894f2c8209b85a2f26307f P 4e1b114af8e445659ca8ca73b2dc1817: Bootstrap starting.
I20260929 18:36:32.340175 16720 tablet_bootstrap.cc:492] T b440bcfdc1894f2c8209b85a2f26307f P 35271bfc58544ae6b541e8d1d99dd3af: Bootstrap starting.
I20260929 18:36:32.340754 16718 tablet_bootstrap.cc:654] T b440bcfdc1894f2c8209b85a2f26307f P 4e1b114af8e445659ca8ca73b2dc1817: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:32.340893 16717 tablet_bootstrap.cc:492] T b440bcfdc1894f2c8209b85a2f26307f P c62d9465adfe4a9ca6c13b227a3726ef: No bootstrap required, opened a new log
I20260929 18:36:32.341091 16717 ts_tablet_manager.cc:1397] T b440bcfdc1894f2c8209b85a2f26307f P c62d9465adfe4a9ca6c13b227a3726ef: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260929 18:36:32.342130 16718 log.cc:824] T b440bcfdc1894f2c8209b85a2f26307f P 4e1b114af8e445659ca8ca73b2dc1817: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:32.342648 16720 tablet_bootstrap.cc:654] T b440bcfdc1894f2c8209b85a2f26307f P 35271bfc58544ae6b541e8d1d99dd3af: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:32.343572 16720 log.cc:824] T b440bcfdc1894f2c8209b85a2f26307f P 35271bfc58544ae6b541e8d1d99dd3af: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:32.345314 16717 raft_consensus.cc:348] T b440bcfdc1894f2c8209b85a2f26307f P c62d9465adfe4a9ca6c13b227a3726ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e1b114af8e445659ca8ca73b2dc1817" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 43137 } } peers { permanent_uuid: "35271bfc58544ae6b541e8d1d99dd3af" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 36945 } } peers { permanent_uuid: "c62d9465adfe4a9ca6c13b227a3726ef" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 37137 } }
I20260929 18:36:32.345613 16717 raft_consensus.cc:374] T b440bcfdc1894f2c8209b85a2f26307f P c62d9465adfe4a9ca6c13b227a3726ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:32.345676 16717 raft_consensus.cc:729] T b440bcfdc1894f2c8209b85a2f26307f P c62d9465adfe4a9ca6c13b227a3726ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c62d9465adfe4a9ca6c13b227a3726ef, State: Initialized, Role: FOLLOWER
I20260929 18:36:32.345829 16718 tablet_bootstrap.cc:492] T b440bcfdc1894f2c8209b85a2f26307f P 4e1b114af8e445659ca8ca73b2dc1817: No bootstrap required, opened a new log
I20260929 18:36:32.346041 16718 ts_tablet_manager.cc:1397] T b440bcfdc1894f2c8209b85a2f26307f P 4e1b114af8e445659ca8ca73b2dc1817: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 18:36:32.346217 16717 consensus_queue.cc:260] T b440bcfdc1894f2c8209b85a2f26307f P c62d9465adfe4a9ca6c13b227a3726ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e1b114af8e445659ca8ca73b2dc1817" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 43137 } } peers { permanent_uuid: "35271bfc58544ae6b541e8d1d99dd3af" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 36945 } } peers { permanent_uuid: "c62d9465adfe4a9ca6c13b227a3726ef" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 37137 } }
I20260929 18:36:32.347218 16566 heartbeater.cc:503] Master 127.28.207.126:33309 was elected leader, sending a full tablet report...
I20260929 18:36:32.347343 16717 ts_tablet_manager.cc:1428] T b440bcfdc1894f2c8209b85a2f26307f P c62d9465adfe4a9ca6c13b227a3726ef: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260929 18:36:32.348829 16720 tablet_bootstrap.cc:492] T b440bcfdc1894f2c8209b85a2f26307f P 35271bfc58544ae6b541e8d1d99dd3af: No bootstrap required, opened a new log
I20260929 18:36:32.349048 16720 ts_tablet_manager.cc:1397] T b440bcfdc1894f2c8209b85a2f26307f P 35271bfc58544ae6b541e8d1d99dd3af: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20260929 18:36:32.350102 16718 raft_consensus.cc:348] T b440bcfdc1894f2c8209b85a2f26307f P 4e1b114af8e445659ca8ca73b2dc1817 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e1b114af8e445659ca8ca73b2dc1817" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 43137 } } peers { permanent_uuid: "35271bfc58544ae6b541e8d1d99dd3af" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 36945 } } peers { permanent_uuid: "c62d9465adfe4a9ca6c13b227a3726ef" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 37137 } }
I20260929 18:36:32.350369 16718 raft_consensus.cc:374] T b440bcfdc1894f2c8209b85a2f26307f P 4e1b114af8e445659ca8ca73b2dc1817 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:32.350419 16718 raft_consensus.cc:729] T b440bcfdc1894f2c8209b85a2f26307f P 4e1b114af8e445659ca8ca73b2dc1817 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e1b114af8e445659ca8ca73b2dc1817, State: Initialized, Role: FOLLOWER
I20260929 18:36:32.350927 16718 consensus_queue.cc:260] T b440bcfdc1894f2c8209b85a2f26307f P 4e1b114af8e445659ca8ca73b2dc1817 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e1b114af8e445659ca8ca73b2dc1817" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 43137 } } peers { permanent_uuid: "35271bfc58544ae6b541e8d1d99dd3af" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 36945 } } peers { permanent_uuid: "c62d9465adfe4a9ca6c13b227a3726ef" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 37137 } }
I20260929 18:36:32.351843 16700 heartbeater.cc:503] Master 127.28.207.126:33309 was elected leader, sending a full tablet report...
I20260929 18:36:32.352051 16718 ts_tablet_manager.cc:1428] T b440bcfdc1894f2c8209b85a2f26307f P 4e1b114af8e445659ca8ca73b2dc1817: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 18:36:32.353598 16720 raft_consensus.cc:348] T b440bcfdc1894f2c8209b85a2f26307f P 35271bfc58544ae6b541e8d1d99dd3af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e1b114af8e445659ca8ca73b2dc1817" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 43137 } } peers { permanent_uuid: "35271bfc58544ae6b541e8d1d99dd3af" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 36945 } } peers { permanent_uuid: "c62d9465adfe4a9ca6c13b227a3726ef" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 37137 } }
I20260929 18:36:32.353891 16720 raft_consensus.cc:374] T b440bcfdc1894f2c8209b85a2f26307f P 35271bfc58544ae6b541e8d1d99dd3af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:32.353945 16720 raft_consensus.cc:729] T b440bcfdc1894f2c8209b85a2f26307f P 35271bfc58544ae6b541e8d1d99dd3af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35271bfc58544ae6b541e8d1d99dd3af, State: Initialized, Role: FOLLOWER
I20260929 18:36:32.354420 16720 consensus_queue.cc:260] T b440bcfdc1894f2c8209b85a2f26307f P 35271bfc58544ae6b541e8d1d99dd3af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e1b114af8e445659ca8ca73b2dc1817" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 43137 } } peers { permanent_uuid: "35271bfc58544ae6b541e8d1d99dd3af" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 36945 } } peers { permanent_uuid: "c62d9465adfe4a9ca6c13b227a3726ef" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 37137 } }
I20260929 18:36:32.355247 16432 heartbeater.cc:503] Master 127.28.207.126:33309 was elected leader, sending a full tablet report...
I20260929 18:36:32.355492 16720 ts_tablet_manager.cc:1428] T b440bcfdc1894f2c8209b85a2f26307f P 35271bfc58544ae6b541e8d1d99dd3af: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
I20260929 18:36:32.435192 16724 raft_consensus.cc:482] T b440bcfdc1894f2c8209b85a2f26307f P 4e1b114af8e445659ca8ca73b2dc1817 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:36:32.435710 16724 raft_consensus.cc:504] T b440bcfdc1894f2c8209b85a2f26307f P 4e1b114af8e445659ca8ca73b2dc1817 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e1b114af8e445659ca8ca73b2dc1817" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 43137 } } peers { permanent_uuid: "35271bfc58544ae6b541e8d1d99dd3af" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 36945 } } peers { permanent_uuid: "c62d9465adfe4a9ca6c13b227a3726ef" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 37137 } }
I20260929 18:36:32.437295 16724 leader_election.cc:288] T b440bcfdc1894f2c8209b85a2f26307f P 4e1b114af8e445659ca8ca73b2dc1817 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35271bfc58544ae6b541e8d1d99dd3af (127.28.207.65:36945), c62d9465adfe4a9ca6c13b227a3726ef (127.28.207.66:37137)
I20260929 18:36:32.440591 16723 raft_consensus.cc:482] T b440bcfdc1894f2c8209b85a2f26307f P c62d9465adfe4a9ca6c13b227a3726ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:36:32.441259 16723 raft_consensus.cc:504] T b440bcfdc1894f2c8209b85a2f26307f P c62d9465adfe4a9ca6c13b227a3726ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e1b114af8e445659ca8ca73b2dc1817" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 43137 } } peers { permanent_uuid: "35271bfc58544ae6b541e8d1d99dd3af" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 36945 } } peers { permanent_uuid: "c62d9465adfe4a9ca6c13b227a3726ef" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 37137 } }
I20260929 18:36:32.442553 16702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:32.437573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:36945 (local address 127.28.207.67:35603)
0929 18:36:32.437773 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:32.437791 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:32.437914 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:32.438447 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:32.438451 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:32.438473 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:32.438618 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:32.438624 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.439715 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.439718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:32.441012 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:32.441024 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.442035 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.442039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:32.442161 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:32.442189 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 18:36:32.442299 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 18:36:32.442389 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260929 18:36:32.442553 16716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:32.437987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:35603 (local address 127.28.207.65:36945)
0929 18:36:32.438131 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:32.438136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:32.438153 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:36:32.438226 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:32.438229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:32.438295 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:32.438404 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:32.438749 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.439558 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.441223 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.441887 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.442050 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:32.442120 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:32.442324 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 18:36:32.442415 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 18:36:32.442888 16714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:32.437845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:36265 (local address 127.28.207.66:37137)
0929 18:36:32.437997 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:32.438003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:32.438369 (+ 366us) server_negotiation.cc:407] Connection header received
0929 18:36:32.438565 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:32.438571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:32.438629 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:32.438698 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:32.439339 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.439974 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.441439 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.442050 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.442204 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:32.442263 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:32.442620 (+ 357us) server_negotiation.cc:299] Negotiation successful
0929 18:36:32.442716 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 18:36:32.443049 16723 leader_election.cc:288] T b440bcfdc1894f2c8209b85a2f26307f P c62d9465adfe4a9ca6c13b227a3726ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e1b114af8e445659ca8ca73b2dc1817 (127.28.207.67:43137), 35271bfc58544ae6b541e8d1d99dd3af (127.28.207.65:36945)
I20260929 18:36:32.443325 16726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:32.437747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:37137 (local address 127.28.207.67:36265)
0929 18:36:32.438266 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:32.438277 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:32.438394 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:32.438914 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:32.438916 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:32.438935 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:32.439147 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:32.439178 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.440114 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.440118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:32.441288 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:32.441299 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.442341 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.442346 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:32.442463 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:32.442491 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 18:36:32.443040 (+ 549us) client_negotiation.cc:240] Negotiation successful
0929 18:36:32.443148 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":182,"threads_started":1}
I20260929 18:36:32.443439 16386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b440bcfdc1894f2c8209b85a2f26307f" candidate_uuid: "4e1b114af8e445659ca8ca73b2dc1817" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35271bfc58544ae6b541e8d1d99dd3af" is_pre_election: true
I20260929 18:36:32.443852 16386 raft_consensus.cc:2454] T b440bcfdc1894f2c8209b85a2f26307f P 35271bfc58544ae6b541e8d1d99dd3af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e1b114af8e445659ca8ca73b2dc1817 in term 0.
I20260929 18:36:32.444734 16520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b440bcfdc1894f2c8209b85a2f26307f" candidate_uuid: "4e1b114af8e445659ca8ca73b2dc1817" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c62d9465adfe4a9ca6c13b227a3726ef" is_pre_election: true
I20260929 18:36:32.444840 16588 leader_election.cc:302] T b440bcfdc1894f2c8209b85a2f26307f P 4e1b114af8e445659ca8ca73b2dc1817 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35271bfc58544ae6b541e8d1d99dd3af, 4e1b114af8e445659ca8ca73b2dc1817; no voters:
I20260929 18:36:32.444994 16520 raft_consensus.cc:2454] T b440bcfdc1894f2c8209b85a2f26307f P c62d9465adfe4a9ca6c13b227a3726ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e1b114af8e445659ca8ca73b2dc1817 in term 0.
I20260929 18:36:32.445214 16724 raft_consensus.cc:2781] T b440bcfdc1894f2c8209b85a2f26307f P 4e1b114af8e445659ca8ca73b2dc1817 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:36:32.445319 16724 raft_consensus.cc:482] T b440bcfdc1894f2c8209b85a2f26307f P 4e1b114af8e445659ca8ca73b2dc1817 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:36:32.445367 16724 raft_consensus.cc:3037] T b440bcfdc1894f2c8209b85a2f26307f P 4e1b114af8e445659ca8ca73b2dc1817 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:32.446810 16724 raft_consensus.cc:504] T b440bcfdc1894f2c8209b85a2f26307f P 4e1b114af8e445659ca8ca73b2dc1817 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e1b114af8e445659ca8ca73b2dc1817" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 43137 } } peers { permanent_uuid: "35271bfc58544ae6b541e8d1d99dd3af" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 36945 } } peers { permanent_uuid: "c62d9465adfe4a9ca6c13b227a3726ef" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 37137 } }
I20260929 18:36:32.447324 16724 leader_election.cc:288] T b440bcfdc1894f2c8209b85a2f26307f P 4e1b114af8e445659ca8ca73b2dc1817 [CANDIDATE]: Term 1 election: Requested vote from peers 35271bfc58544ae6b541e8d1d99dd3af (127.28.207.65:36945), c62d9465adfe4a9ca6c13b227a3726ef (127.28.207.66:37137)
I20260929 18:36:32.447427 16568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:32.443208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:43137 (local address 127.28.207.66:46249)
0929 18:36:32.443603 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:32.443620 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:32.443742 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:32.444038 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:32.444042 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:32.444071 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:32.444233 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:32.444244 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.445117 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.445124 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:32.446082 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:32.446092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.446843 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.446847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:32.447008 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:32.447031 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 18:36:32.447152 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 18:36:32.447241 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283}
I20260929 18:36:32.447533 16712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:32.443320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:46249 (local address 127.28.207.67:43137)
0929 18:36:32.443456 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:32.443461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:32.443692 (+ 231us) server_negotiation.cc:407] Connection header received
0929 18:36:32.443861 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:32.443865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:32.443919 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:32.443998 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:32.444404 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.444983 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.446229 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.446688 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.446882 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:32.446946 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:32.447191 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 18:36:32.447299 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 18:36:32.447777 16520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b440bcfdc1894f2c8209b85a2f26307f" candidate_uuid: "4e1b114af8e445659ca8ca73b2dc1817" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c62d9465adfe4a9ca6c13b227a3726ef"
I20260929 18:36:32.447800 16386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b440bcfdc1894f2c8209b85a2f26307f" candidate_uuid: "4e1b114af8e445659ca8ca73b2dc1817" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35271bfc58544ae6b541e8d1d99dd3af"
I20260929 18:36:32.447908 16520 raft_consensus.cc:3037] T b440bcfdc1894f2c8209b85a2f26307f P c62d9465adfe4a9ca6c13b227a3726ef [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:32.447973 16386 raft_consensus.cc:3037] T b440bcfdc1894f2c8209b85a2f26307f P 35271bfc58544ae6b541e8d1d99dd3af [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:32.448382 16654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b440bcfdc1894f2c8209b85a2f26307f" candidate_uuid: "c62d9465adfe4a9ca6c13b227a3726ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e1b114af8e445659ca8ca73b2dc1817" is_pre_election: true
I20260929 18:36:32.448680 16654 raft_consensus.cc:2379] T b440bcfdc1894f2c8209b85a2f26307f P 4e1b114af8e445659ca8ca73b2dc1817 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c62d9465adfe4a9ca6c13b227a3726ef in current term 1: Already voted for candidate 4e1b114af8e445659ca8ca73b2dc1817 in this term.
I20260929 18:36:32.449520 16520 raft_consensus.cc:2454] T b440bcfdc1894f2c8209b85a2f26307f P c62d9465adfe4a9ca6c13b227a3726ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e1b114af8e445659ca8ca73b2dc1817 in term 1.
I20260929 18:36:32.449676 16386 raft_consensus.cc:2454] T b440bcfdc1894f2c8209b85a2f26307f P 35271bfc58544ae6b541e8d1d99dd3af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e1b114af8e445659ca8ca73b2dc1817 in term 1.
I20260929 18:36:32.449918 16590 leader_election.cc:302] T b440bcfdc1894f2c8209b85a2f26307f P 4e1b114af8e445659ca8ca73b2dc1817 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e1b114af8e445659ca8ca73b2dc1817, c62d9465adfe4a9ca6c13b227a3726ef; no voters:
I20260929 18:36:32.450214 16724 raft_consensus.cc:2781] T b440bcfdc1894f2c8209b85a2f26307f P 4e1b114af8e445659ca8ca73b2dc1817 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:32.450485 16724 raft_consensus.cc:686] T b440bcfdc1894f2c8209b85a2f26307f P 4e1b114af8e445659ca8ca73b2dc1817 [term 1 LEADER]: Becoming Leader. State: Replica: 4e1b114af8e445659ca8ca73b2dc1817, State: Running, Role: LEADER
I20260929 18:36:32.451050 16724 consensus_queue.cc:237] T b440bcfdc1894f2c8209b85a2f26307f P 4e1b114af8e445659ca8ca73b2dc1817 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e1b114af8e445659ca8ca73b2dc1817" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 43137 } } peers { permanent_uuid: "35271bfc58544ae6b541e8d1d99dd3af" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 36945 } } peers { permanent_uuid: "c62d9465adfe4a9ca6c13b227a3726ef" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 37137 } }
I20260929 18:36:32.454571 16568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:32.443591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:36945 (local address 127.28.207.66:52847)
0929 18:36:32.447642 (+ 4051us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:32.447659 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:32.447758 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:32.448685 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:32.448689 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:32.448707 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:32.448801 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:32.448806 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.449868 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.449872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:32.450826 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:32.450836 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.454082 (+ 3246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.454086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:32.454181 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:32.454212 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 18:36:32.454320 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 18:36:32.454402 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3986,"thread_start_us":164,"threads_started":1}
I20260929 18:36:32.454686 16716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:32.443939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:52847 (local address 127.28.207.65:36945)
0929 18:36:32.444074 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:32.444081 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:32.448278 (+ 4197us) server_negotiation.cc:407] Connection header received
0929 18:36:32.448351 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:32.448354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:32.448410 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:32.448493 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:32.448989 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.449757 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.453075 (+ 3318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.453909 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.454066 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:32.454139 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:32.454353 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 18:36:32.454441 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 18:36:32.455268 16386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b440bcfdc1894f2c8209b85a2f26307f" candidate_uuid: "c62d9465adfe4a9ca6c13b227a3726ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35271bfc58544ae6b541e8d1d99dd3af" is_pre_election: true
I20260929 18:36:32.455515 16386 raft_consensus.cc:2379] T b440bcfdc1894f2c8209b85a2f26307f P 35271bfc58544ae6b541e8d1d99dd3af [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c62d9465adfe4a9ca6c13b227a3726ef in current term 1: Already voted for candidate 4e1b114af8e445659ca8ca73b2dc1817 in this term.
I20260929 18:36:32.455679 16238 catalog_manager.cc:5495] T b440bcfdc1894f2c8209b85a2f26307f P 4e1b114af8e445659ca8ca73b2dc1817 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e1b114af8e445659ca8ca73b2dc1817 (127.28.207.67). New cstate: current_term: 1 leader_uuid: "4e1b114af8e445659ca8ca73b2dc1817" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e1b114af8e445659ca8ca73b2dc1817" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 43137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35271bfc58544ae6b541e8d1d99dd3af" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 36945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c62d9465adfe4a9ca6c13b227a3726ef" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 37137 } health_report { overall_health: UNKNOWN } } }
I20260929 18:36:32.456043 16454 leader_election.cc:302] T b440bcfdc1894f2c8209b85a2f26307f P c62d9465adfe4a9ca6c13b227a3726ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c62d9465adfe4a9ca6c13b227a3726ef; no voters: 35271bfc58544ae6b541e8d1d99dd3af, 4e1b114af8e445659ca8ca73b2dc1817
I20260929 18:36:32.456364 16723 raft_consensus.cc:2726] T b440bcfdc1894f2c8209b85a2f26307f P c62d9465adfe4a9ca6c13b227a3726ef [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 18:36:32.479959 16712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:32.475155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41448 (local address 127.28.207.67:43137)
0929 18:36:32.475368 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:32.475373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:32.475391 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:36:32.475665 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:32.475667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:32.475743 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:32.475949 (+ 206us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:36:32.476309 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.476931 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.478056 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.478286 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.478403 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:32.478768 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:36:32.479426 (+ 658us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:36:32.479560 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:32.479617 (+ 57us) server_negotiation.cc:299] Negotiation successful
0929 18:36:32.479681 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260929 18:36:32.487185 16433 tablet.cc:2369] T b440bcfdc1894f2c8209b85a2f26307f P 35271bfc58544ae6b541e8d1d99dd3af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 18:36:32.487457 16386 raft_consensus.cc:1261] T b440bcfdc1894f2c8209b85a2f26307f P 35271bfc58544ae6b541e8d1d99dd3af [term 1 FOLLOWER]: Refusing update from remote peer 4e1b114af8e445659ca8ca73b2dc1817: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:36:32.487869 16520 raft_consensus.cc:1261] T b440bcfdc1894f2c8209b85a2f26307f P c62d9465adfe4a9ca6c13b227a3726ef [term 1 FOLLOWER]: Refusing update from remote peer 4e1b114af8e445659ca8ca73b2dc1817: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:36:32.488252 16728 consensus_queue.cc:1035] T b440bcfdc1894f2c8209b85a2f26307f P 4e1b114af8e445659ca8ca73b2dc1817 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35271bfc58544ae6b541e8d1d99dd3af" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 36945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:36:32.488616 16724 consensus_queue.cc:1035] T b440bcfdc1894f2c8209b85a2f26307f P 4e1b114af8e445659ca8ca73b2dc1817 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c62d9465adfe4a9ca6c13b227a3726ef" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 37137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:36:32.498485 16732 mvcc.cc:205] Tried to move back new op lower bound from 7334735841219112960 to 7334735841082867712. Current Snapshot: MvccSnapshot[applied={T|T < 7334735841219112960}]
I20260929 18:36:32.499881 16733 mvcc.cc:205] Tried to move back new op lower bound from 7334735841219112960 to 7334735841082867712. Current Snapshot: MvccSnapshot[applied={T|T < 7334735841219112960}]
I20260929 18:36:32.507848 16240 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:36:32.508037 16240 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:36:32.510437 16240 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a1322574edd44567ba375a4c1dab5d8e: Sending DeleteTablet for 3 replicas of tablet b440bcfdc1894f2c8209b85a2f26307f
I20260929 18:36:32.511608 16496 tablet_service.cc:1514] Processing DeleteTablet for tablet b440bcfdc1894f2c8209b85a2f26307f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:32 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:33498
I20260929 18:36:32.511605 16366 tablet_service.cc:1514] Processing DeleteTablet for tablet b440bcfdc1894f2c8209b85a2f26307f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:32 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:49298
I20260929 18:36:32.512161 16741 tablet_replica.cc:330] T b440bcfdc1894f2c8209b85a2f26307f P c62d9465adfe4a9ca6c13b227a3726ef: stopping tablet replica
I20260929 18:36:32.512130 16634 tablet_service.cc:1514] Processing DeleteTablet for tablet b440bcfdc1894f2c8209b85a2f26307f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:32 UTC) from {username='kudu', principal='kudu/127.28.207.126@KRBTEST.COM'} at 127.0.0.1:41442
I20260929 18:36:32.512444 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 16303
I20260929 18:36:32.512537 16741 raft_consensus.cc:2229] T b440bcfdc1894f2c8209b85a2f26307f P c62d9465adfe4a9ca6c13b227a3726ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:36:32.513568 16742 tablet_replica.cc:330] T b440bcfdc1894f2c8209b85a2f26307f P 4e1b114af8e445659ca8ca73b2dc1817: stopping tablet replica
I20260929 18:36:32.513825 16742 raft_consensus.cc:2229] T b440bcfdc1894f2c8209b85a2f26307f P 4e1b114af8e445659ca8ca73b2dc1817 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:36:32.514251 16742 raft_consensus.cc:2258] T b440bcfdc1894f2c8209b85a2f26307f P 4e1b114af8e445659ca8ca73b2dc1817 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:36:32.514525 16741 raft_consensus.cc:2258] T b440bcfdc1894f2c8209b85a2f26307f P c62d9465adfe4a9ca6c13b227a3726ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:36:32.516266 16742 ts_tablet_manager.cc:1905] T b440bcfdc1894f2c8209b85a2f26307f P 4e1b114af8e445659ca8ca73b2dc1817: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:36:32.519484 16741 ts_tablet_manager.cc:1905] T b440bcfdc1894f2c8209b85a2f26307f P c62d9465adfe4a9ca6c13b227a3726ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:36:32.519945 16742 ts_tablet_manager.cc:1918] T b440bcfdc1894f2c8209b85a2f26307f P 4e1b114af8e445659ca8ca73b2dc1817: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:36:32.520047 16742 log.cc:1196] T b440bcfdc1894f2c8209b85a2f26307f P 4e1b114af8e445659ca8ca73b2dc1817: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-2/wal/wals/b440bcfdc1894f2c8209b85a2f26307f
I20260929 18:36:32.520422 16742 ts_tablet_manager.cc:1939] T b440bcfdc1894f2c8209b85a2f26307f P 4e1b114af8e445659ca8ca73b2dc1817: Deleting consensus metadata
I20260929 18:36:32.521826 16226 catalog_manager.cc:4841] TS 4e1b114af8e445659ca8ca73b2dc1817 (127.28.207.67:43137): tablet b440bcfdc1894f2c8209b85a2f26307f (table test-table [id=c0bddcd6b8f8401fb9d499a742c812a7]) successfully deleted
I20260929 18:36:32.523012 16741 ts_tablet_manager.cc:1918] T b440bcfdc1894f2c8209b85a2f26307f P c62d9465adfe4a9ca6c13b227a3726ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:36:32.523164 16741 log.cc:1196] T b440bcfdc1894f2c8209b85a2f26307f P c62d9465adfe4a9ca6c13b227a3726ef: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790706718260899-29501-0/minicluster-data/ts-1/wal/wals/b440bcfdc1894f2c8209b85a2f26307f
W20260929 18:36:32.523227 16226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.207.65:36945 (error 108)
I20260929 18:36:32.523511 16741 ts_tablet_manager.cc:1939] T b440bcfdc1894f2c8209b85a2f26307f P c62d9465adfe4a9ca6c13b227a3726ef: Deleting consensus metadata
I20260929 18:36:32.523663 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 16437
I20260929 18:36:32.525462 16713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:32.525061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:36945 (local address 127.0.0.1:49304)
0929 18:36:32.525268 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:32.525342 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.65:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260929 18:36:32.525626 16226 catalog_manager.cc:4584] TS 35271bfc58544ae6b541e8d1d99dd3af (127.28.207.65:36945): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b440bcfdc1894f2c8209b85a2f26307f: Network error: Client connection negotiation failed: client connection to 127.28.207.65:36945: connect: Connection refused (error 111)
I20260929 18:36:32.530660 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 16571
I20260929 18:36:32.532578 16713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:32.532286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:37137 (local address 127.0.0.1:33500)
0929 18:36:32.532407 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:32.532471 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.66:37137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 18:36:32.537590 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 16209
2026-09-29T18:36:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2581 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T18:36:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:36:32Z Disabled control of system clock
I20260929 18:36:32.573741 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:42641
--webserver_interface=127.28.207.126
--webserver_port=0
--builtin_ntp_servers=127.28.207.84:45423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:42641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:36:32.719712 16747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:32.720094 16747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:32.720135 16747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:32.724470 16747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:36:32.724547 16747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:32.724565 16747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:32.724582 16747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:36:32.724598 16747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:36:32.730238 16747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:45423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:42641
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:42641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:32.732182 16747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:32.734157 16747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:32.745078 16755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:32.745180 16752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:32.745357 16747 server_base.cc:1034] running on GCE node
W20260929 18:36:32.745092 16753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:32.746519 16747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:32.748145 16747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:32.749380 16747 hybrid_clock.cc:648] HybridClock initialized: now 1790706992749344 us; error 55 us; skew 500 ppm
I20260929 18:36:32.760882 16747 webserver.cc:460] Webserver started at http://127.28.207.126:38729/ using document root <none> and password file <none>
I20260929 18:36:32.761935 16747 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:32.762048 16747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:32.762344 16747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:32.766885 16747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "0dd30edc9a9e4cfd9b58787191a6024a"
format_stamp: "Formatted at 2026-09-29 18:36:32 on dist-test-slave-qcpv"
server_key: "4e8ade1f57d85f72e53e26ec72db0e3c"
server_key_iv: "be7446d017e5284f61f0627c23b39af4"
server_key_version: "encryptionkey@0"
I20260929 18:36:32.768025 16747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "0dd30edc9a9e4cfd9b58787191a6024a"
format_stamp: "Formatted at 2026-09-29 18:36:32 on dist-test-slave-qcpv"
server_key: "4e8ade1f57d85f72e53e26ec72db0e3c"
server_key_iv: "be7446d017e5284f61f0627c23b39af4"
server_key_version: "encryptionkey@0"
I20260929 18:36:32.774920 16747 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260929 18:36:32.779901 16761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:32.782081 16747 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 18:36:32.782321 16747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "0dd30edc9a9e4cfd9b58787191a6024a"
format_stamp: "Formatted at 2026-09-29 18:36:32 on dist-test-slave-qcpv"
server_key: "4e8ade1f57d85f72e53e26ec72db0e3c"
server_key_iv: "be7446d017e5284f61f0627c23b39af4"
server_key_version: "encryptionkey@0"
I20260929 18:36:32.782511 16747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:32.806483 16747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:32.807551 16747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:32.807775 16747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:32.817024 16747 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:42641
I20260929 18:36:32.817024 16813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:42641 every 8 connection(s)
I20260929 18:36:32.818069 16747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:36:32.822630 16814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:32.824084 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 16747
I20260929 18:36:32.824203 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:36:32.824575 29501 external_mini_cluster.cc:1428] Setting key 64a0f4357df27558cf140cc658f12416
I20260929 18:36:32.831554 16814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dd30edc9a9e4cfd9b58787191a6024a: Bootstrap starting.
I20260929 18:36:32.835315 16814 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0dd30edc9a9e4cfd9b58787191a6024a: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:32.836047 16817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:32.826125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60354 (local address 127.28.207.126:42641)
0929 18:36:32.826909 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:32.826932 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:32.826985 (+ 53us) server_negotiation.cc:407] Connection header received
0929 18:36:32.828060 (+ 1075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:32.828082 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:32.828540 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:32.828853 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:32.829416 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.830482 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.831914 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:32.832472 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:32.832622 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:32.833815 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:32.833823 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:32.833845 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:32.834293 (+ 448us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:32.834372 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:32.834395 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:32.834497 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:32.834697 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 18:36:32.835103 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":144,"threads_started":1}
I20260929 18:36:32.836360 16814 log.cc:824] T 00000000000000000000000000000000 P 0dd30edc9a9e4cfd9b58787191a6024a: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:32.838788 16814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dd30edc9a9e4cfd9b58787191a6024a: No bootstrap required, opened a new log
I20260929 18:36:32.842839 16814 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0dd30edc9a9e4cfd9b58787191a6024a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dd30edc9a9e4cfd9b58787191a6024a" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 42641 } }
I20260929 18:36:32.843166 16814 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0dd30edc9a9e4cfd9b58787191a6024a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:32.843205 16814 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0dd30edc9a9e4cfd9b58787191a6024a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dd30edc9a9e4cfd9b58787191a6024a, State: Initialized, Role: FOLLOWER
I20260929 18:36:32.843642 16814 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0dd30edc9a9e4cfd9b58787191a6024a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dd30edc9a9e4cfd9b58787191a6024a" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 42641 } }
I20260929 18:36:32.843780 16814 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0dd30edc9a9e4cfd9b58787191a6024a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:36:32.844067 16814 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0dd30edc9a9e4cfd9b58787191a6024a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:36:32.844199 16814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0dd30edc9a9e4cfd9b58787191a6024a [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:32.845257 16814 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0dd30edc9a9e4cfd9b58787191a6024a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dd30edc9a9e4cfd9b58787191a6024a" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 42641 } }
I20260929 18:36:32.845566 16814 leader_election.cc:302] T 00000000000000000000000000000000 P 0dd30edc9a9e4cfd9b58787191a6024a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0dd30edc9a9e4cfd9b58787191a6024a; no voters:
I20260929 18:36:32.845851 16814 leader_election.cc:288] T 00000000000000000000000000000000 P 0dd30edc9a9e4cfd9b58787191a6024a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:36:32.846181 16819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0dd30edc9a9e4cfd9b58787191a6024a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:32.846520 16819 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0dd30edc9a9e4cfd9b58787191a6024a [term 1 LEADER]: Becoming Leader. State: Replica: 0dd30edc9a9e4cfd9b58787191a6024a, State: Running, Role: LEADER
I20260929 18:36:32.847052 16819 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0dd30edc9a9e4cfd9b58787191a6024a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dd30edc9a9e4cfd9b58787191a6024a" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 42641 } }
I20260929 18:36:32.847540 16814 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0dd30edc9a9e4cfd9b58787191a6024a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:36:32.849030 16820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dd30edc9a9e4cfd9b58787191a6024a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0dd30edc9a9e4cfd9b58787191a6024a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dd30edc9a9e4cfd9b58787191a6024a" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 42641 } } }
I20260929 18:36:32.849195 16820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dd30edc9a9e4cfd9b58787191a6024a [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:32.849573 16821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dd30edc9a9e4cfd9b58787191a6024a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0dd30edc9a9e4cfd9b58787191a6024a. Latest consensus state: current_term: 1 leader_uuid: "0dd30edc9a9e4cfd9b58787191a6024a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dd30edc9a9e4cfd9b58787191a6024a" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 42641 } } }
I20260929 18:36:32.849807 16821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dd30edc9a9e4cfd9b58787191a6024a [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:32.849731 16829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:36:32.854125 16829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:36:32.861819 16829 catalog_manager.cc:1347] Generated new cluster ID: a83a36e0c48540f4a29986f05740f1c6
I20260929 18:36:32.861946 16829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:36:32.873189 16829 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:36:32.874352 16829 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:36:32.888873 16829 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0dd30edc9a9e4cfd9b58787191a6024a: Generated new TSK 0
I20260929 18:36:32.889838 16829 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:36:32.906762 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.65:0
--local_ip_for_outbound_sockets=127.28.207.65
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:42641
--builtin_ntp_servers=127.28.207.84:45423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 18:36:33.030553 16838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:33.030900 16838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:33.030942 16838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:33.035171 16838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:33.035243 16838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:33.035302 16838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.65
I20260929 18:36:33.040213 16838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:45423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.28.207.65
--webserver_port=0
--tserver_master_addrs=127.28.207.126:42641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.65
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:33.041432 16838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:33.042891 16838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:33.053591 16846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:33.053591 16844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:33.053625 16843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:33.053900 16838 server_base.cc:1034] running on GCE node
I20260929 18:36:33.054576 16838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:33.055661 16838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:33.056928 16838 hybrid_clock.cc:648] HybridClock initialized: now 1790706993056884 us; error 66 us; skew 500 ppm
I20260929 18:36:33.059881 16838 webserver.cc:460] Webserver started at http://127.28.207.65:40521/ using document root <none> and password file <none>
I20260929 18:36:33.060765 16838 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:33.060856 16838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:33.061072 16838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:33.063688 16838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-0/data/instance:
uuid: "6cb485f27cfd4426b66edc0db974ebb8"
format_stamp: "Formatted at 2026-09-29 18:36:33 on dist-test-slave-qcpv"
server_key: "45007d3a79f1e404e3a37864466f74b6"
server_key_iv: "863a4df40db6b7c7b63394c7664f70cc"
server_key_version: "encryptionkey@0"
I20260929 18:36:33.064296 16838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance:
uuid: "6cb485f27cfd4426b66edc0db974ebb8"
format_stamp: "Formatted at 2026-09-29 18:36:33 on dist-test-slave-qcpv"
server_key: "45007d3a79f1e404e3a37864466f74b6"
server_key_iv: "863a4df40db6b7c7b63394c7664f70cc"
server_key_version: "encryptionkey@0"
I20260929 18:36:33.069268 16838 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 18:36:33.073364 16852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:33.075486 16838 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 18:36:33.075729 16838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-0/wal
uuid: "6cb485f27cfd4426b66edc0db974ebb8"
format_stamp: "Formatted at 2026-09-29 18:36:33 on dist-test-slave-qcpv"
server_key: "45007d3a79f1e404e3a37864466f74b6"
server_key_iv: "863a4df40db6b7c7b63394c7664f70cc"
server_key_version: "encryptionkey@0"
I20260929 18:36:33.075901 16838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:33.101764 16838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:33.102701 16838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:33.102989 16838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:33.103730 16838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:33.104925 16838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:33.105005 16838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:33.105088 16838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:33.105134 16838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:33.117110 16838 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.65:33073
I20260929 18:36:33.117132 16965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.65:33073 every 8 connection(s)
I20260929 18:36:33.118443 16838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-0/data/info.pb
I20260929 18:36:33.124747 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 16838
I20260929 18:36:33.124913 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-0/wal/instance
I20260929 18:36:33.125289 29501 external_mini_cluster.cc:1428] Setting key 6f2a571053dbce2ec989524e6c455e9c
I20260929 18:36:33.127861 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.66:0
--local_ip_for_outbound_sockets=127.28.207.66
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:42641
--builtin_ntp_servers=127.28.207.84:45423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 18:36:33.132270 16817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:33.121717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.65:50789 (local address 127.28.207.126:42641)
0929 18:36:33.121960 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:33.121964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:33.123578 (+ 1614us) server_negotiation.cc:407] Connection header received
0929 18:36:33.125705 (+ 2127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:33.125710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:33.125831 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:33.125987 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:33.127334 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.128170 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.130036 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.130311 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.130457 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:33.131721 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:33.131725 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:33.131727 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:33.131749 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:33.131767 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:33.131775 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:33.131852 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:33.132036 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 18:36:33.132115 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260929 18:36:33.133292 16968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:33.122368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:42641 (local address 127.28.207.65:50789)
0929 18:36:33.123385 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:33.123441 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:33.125349 (+ 1908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:33.126291 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:33.126300 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:33.126402 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:33.127126 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:33.127144 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.128338 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.128345 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:33.129897 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:33.129910 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.130511 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.130515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:33.130698 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:33.131432 (+ 734us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:36:33.131434 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:33.131478 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:36:33.131550 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:33.131867 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:33.131877 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:33.131883 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:36:33.132021 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 18:36:33.132346 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":177,"threads_started":1}
I20260929 18:36:33.135341 16966 heartbeater.cc:347] Connected to a master server at 127.28.207.126:42641
I20260929 18:36:33.135653 16966 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:33.136217 16966 heartbeater.cc:511] Master 127.28.207.126:42641 requested a full tablet report, sending...
I20260929 18:36:33.138823 16775 ts_manager.cc:194] Registered new tserver with Master: 6cb485f27cfd4426b66edc0db974ebb8 (127.28.207.65:33073)
I20260929 18:36:33.141537 16775 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.207.65:50789
W20260929 18:36:33.269344 16969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:33.269694 16969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:33.269737 16969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:33.274824 16969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:33.274974 16969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:33.275120 16969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.66
I20260929 18:36:33.284262 16969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:45423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.28.207.66
--webserver_port=0
--tserver_master_addrs=127.28.207.126:42641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.66
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:33.285723 16969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:33.287590 16969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:33.299666 16975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:33.299831 16969 server_base.cc:1034] running on GCE node
W20260929 18:36:33.299856 16977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:33.299618 16974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:33.300477 16969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:33.301227 16969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:33.302474 16969 hybrid_clock.cc:648] HybridClock initialized: now 1790706993302458 us; error 42 us; skew 500 ppm
I20260929 18:36:33.304821 16969 webserver.cc:460] Webserver started at http://127.28.207.66:42459/ using document root <none> and password file <none>
I20260929 18:36:33.305554 16969 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:33.305656 16969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:33.305956 16969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:33.308543 16969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-1/data/instance:
uuid: "d058fe2e059b4412a7242e7ac6ebc571"
format_stamp: "Formatted at 2026-09-29 18:36:33 on dist-test-slave-qcpv"
server_key: "19c1ada41bcacff4e7744c03d978f553"
server_key_iv: "7076a83beafd436643d6871ed8a6d967"
server_key_version: "encryptionkey@0"
I20260929 18:36:33.309201 16969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance:
uuid: "d058fe2e059b4412a7242e7ac6ebc571"
format_stamp: "Formatted at 2026-09-29 18:36:33 on dist-test-slave-qcpv"
server_key: "19c1ada41bcacff4e7744c03d978f553"
server_key_iv: "7076a83beafd436643d6871ed8a6d967"
server_key_version: "encryptionkey@0"
I20260929 18:36:33.314333 16969 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260929 18:36:33.317795 16983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:33.319716 16969 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 18:36:33.319927 16969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-1/wal
uuid: "d058fe2e059b4412a7242e7ac6ebc571"
format_stamp: "Formatted at 2026-09-29 18:36:33 on dist-test-slave-qcpv"
server_key: "19c1ada41bcacff4e7744c03d978f553"
server_key_iv: "7076a83beafd436643d6871ed8a6d967"
server_key_version: "encryptionkey@0"
I20260929 18:36:33.320132 16969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:33.353222 16969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:33.354103 16969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:33.354283 16969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:33.355028 16969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:33.356180 16969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:33.356232 16969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:33.356281 16969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:33.356297 16969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:33.368423 16969 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.66:34033
I20260929 18:36:33.368642 17096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.66:34033 every 8 connection(s)
I20260929 18:36:33.369513 16969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-1/data/info.pb
I20260929 18:36:33.377895 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 16969
I20260929 18:36:33.378022 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-1/wal/instance
I20260929 18:36:33.378311 29501 external_mini_cluster.cc:1428] Setting key 33eb878e31e0e5decd5e6629f352df79
I20260929 18:36:33.380291 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.28.207.67:0
--local_ip_for_outbound_sockets=127.28.207.67
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:42641
--builtin_ntp_servers=127.28.207.84:45423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 18:36:33.381636 16817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:33.372150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:49397 (local address 127.28.207.126:42641)
0929 18:36:33.372358 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:33.372363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:33.373590 (+ 1227us) server_negotiation.cc:407] Connection header received
0929 18:36:33.375468 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:33.375471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:33.375528 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:33.375639 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:33.376777 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.377348 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.379076 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.379364 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.379505 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:33.381054 (+ 1549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:33.381060 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:33.381068 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:33.381082 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:33.381103 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:33.381112 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:33.381197 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:33.381376 (+ 179us) server_negotiation.cc:299] Negotiation successful
0929 18:36:33.381460 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 18:36:33.382745 17099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:33.372520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:42641 (local address 127.28.207.66:49397)
0929 18:36:33.373377 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:33.373436 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:33.375196 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:33.375885 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:33.375893 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:33.375995 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:33.376621 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:33.376636 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.377567 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.377579 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:33.378800 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:33.378811 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.379563 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.379567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:33.379806 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:33.380690 (+ 884us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:36:33.380692 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:33.380760 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:36:33.380889 (+ 129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:33.381210 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:33.381222 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:33.381228 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 18:36:33.381379 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 18:36:33.381736 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":108,"threads_started":1}
I20260929 18:36:33.384275 17097 heartbeater.cc:347] Connected to a master server at 127.28.207.126:42641
I20260929 18:36:33.384585 17097 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:33.385259 17097 heartbeater.cc:511] Master 127.28.207.126:42641 requested a full tablet report, sending...
I20260929 18:36:33.386567 16775 ts_manager.cc:194] Registered new tserver with Master: d058fe2e059b4412a7242e7ac6ebc571 (127.28.207.66:34033)
I20260929 18:36:33.387696 16775 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.207.66:49397
W20260929 18:36:33.499908 17100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:33.500182 17100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:33.500221 17100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:33.504492 17100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:33.504565 17100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:33.504623 17100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.28.207.67
I20260929 18:36:33.509549 17100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:45423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.28.207.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.28.207.67
--webserver_port=0
--tserver_master_addrs=127.28.207.126:42641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.28.207.67
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:33.510752 17100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:33.512235 17100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:33.523015 17108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:33.523015 17106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:33.523025 17105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:33.523391 17100 server_base.cc:1034] running on GCE node
I20260929 18:36:33.524016 17100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:33.524772 17100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:33.525969 17100 hybrid_clock.cc:648] HybridClock initialized: now 1790706993525948 us; error 47 us; skew 500 ppm
I20260929 18:36:33.528190 17100 webserver.cc:460] Webserver started at http://127.28.207.67:35615/ using document root <none> and password file <none>
I20260929 18:36:33.528831 17100 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:33.528925 17100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:33.529166 17100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:33.531383 17100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-2/data/instance:
uuid: "25dca257cb1f43b5b46922156f2b7cf9"
format_stamp: "Formatted at 2026-09-29 18:36:33 on dist-test-slave-qcpv"
server_key: "5753a6b7d398ace73baf39fabed925d0"
server_key_iv: "e01336a6e95b1534112e779669d14b82"
server_key_version: "encryptionkey@0"
I20260929 18:36:33.531980 17100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance:
uuid: "25dca257cb1f43b5b46922156f2b7cf9"
format_stamp: "Formatted at 2026-09-29 18:36:33 on dist-test-slave-qcpv"
server_key: "5753a6b7d398ace73baf39fabed925d0"
server_key_iv: "e01336a6e95b1534112e779669d14b82"
server_key_version: "encryptionkey@0"
I20260929 18:36:33.536173 17100 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:36:33.539285 17114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:33.540863 17100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:36:33.541035 17100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-2/wal
uuid: "25dca257cb1f43b5b46922156f2b7cf9"
format_stamp: "Formatted at 2026-09-29 18:36:33 on dist-test-slave-qcpv"
server_key: "5753a6b7d398ace73baf39fabed925d0"
server_key_iv: "e01336a6e95b1534112e779669d14b82"
server_key_version: "encryptionkey@0"
I20260929 18:36:33.541172 17100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:33.559719 17100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:33.560500 17100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:33.560736 17100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:33.561431 17100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 18:36:33.562494 17100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 18:36:33.562569 17100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:33.562639 17100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 18:36:33.562683 17100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:33.574249 17100 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.67:45971
I20260929 18:36:33.574280 17227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.67:45971 every 8 connection(s)
I20260929 18:36:33.575310 17100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-2/data/info.pb
I20260929 18:36:33.578366 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 17100
I20260929 18:36:33.578560 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-2/wal/instance
I20260929 18:36:33.578997 29501 external_mini_cluster.cc:1428] Setting key 7d798c9df9b286cd118513d094f30ffa
I20260929 18:36:33.585618 16817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:33.577541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.67:37747 (local address 127.28.207.126:42641)
0929 18:36:33.577825 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:33.577831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:33.578659 (+ 828us) server_negotiation.cc:407] Connection header received
0929 18:36:33.580232 (+ 1573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:33.580235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:33.580293 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:33.580367 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:33.581374 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.582162 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.583544 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.583808 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.583912 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:33.585071 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:33.585074 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:33.585076 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:33.585090 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:33.585114 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:33.585122 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:33.585194 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:33.585380 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 18:36:33.585451 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260929 18:36:33.586442 17230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:33.577729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:42641 (local address 127.28.207.67:37747)
0929 18:36:33.578504 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:33.578555 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:33.580014 (+ 1459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:33.580536 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:33.580542 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:33.580619 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:36:33.581178 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:33.581196 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.582290 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.582293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:33.583413 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:33.583422 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.583954 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.583956 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:33.584134 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:33.584801 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 18:36:33.584802 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:36:33.584841 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 18:36:33.584901 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:36:33.585213 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:36:33.585218 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:36:33.585223 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 18:36:33.585322 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 18:36:33.585537 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":180,"threads_started":1}
I20260929 18:36:33.588001 17228 heartbeater.cc:347] Connected to a master server at 127.28.207.126:42641
I20260929 18:36:33.588248 17228 heartbeater.cc:464] Registering TS with master...
I20260929 18:36:33.588819 17228 heartbeater.cc:511] Master 127.28.207.126:42641 requested a full tablet report, sending...
I20260929 18:36:33.589937 16775 ts_manager.cc:194] Registered new tserver with Master: 25dca257cb1f43b5b46922156f2b7cf9 (127.28.207.67:45971)
I20260929 18:36:33.590512 16775 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.28.207.67:37747
I20260929 18:36:33.594478 29501 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 18:36:33.602021 16817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:33.597564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60356 (local address 127.28.207.126:42641)
0929 18:36:33.597720 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:33.597724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:33.597851 (+ 127us) server_negotiation.cc:407] Connection header received
0929 18:36:33.597986 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:33.597989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:33.598038 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:33.598118 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:33.598523 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.599195 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.600201 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.600433 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.600528 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:33.601504 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:33.601507 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:33.601509 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:33.601523 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:33.601550 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:33.601559 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:33.601628 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:33.601823 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 18:36:33.601889 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 18:36:33.606304 16775 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60356:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 18:36:33.610071 16775 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 18:36:33.629444 17242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:33.623175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:33073 (local address 127.0.0.1:45590)
0929 18:36:33.623407 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:33.623420 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:33.623496 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:33.624394 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:33.624400 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:33.624423 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:33.624571 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:33.624578 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.626541 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.626544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:33.627848 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:33.627858 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.628893 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.628897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:33.629005 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:33.629076 (+ 71us) client_negotiation.cc:769] Sending connection context
0929 18:36:33.629167 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 18:36:33.629303 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":60,"thread_start_us":94,"threads_started":1}
I20260929 18:36:33.629868 17239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:33.621983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:45971 (local address 127.0.0.1:33654)
0929 18:36:33.622787 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:33.622887 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:33.623057 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:33.624238 (+ 1181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:33.624246 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:33.624272 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:33.624462 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:33.624473 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.625875 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.625877 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:33.627893 (+ 2016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:33.627905 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.629476 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.629480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:33.629568 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:33.629586 (+ 18us) client_negotiation.cc:769] Sending connection context
0929 18:36:33.629646 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 18:36:33.629704 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"mutex_wait_us":205,"thread_start_us":92,"threads_started":1}
I20260929 18:36:33.629868 17241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:33.622763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:34033 (local address 127.0.0.1:39172)
0929 18:36:33.623105 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:33.623123 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:33.623229 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:33.624489 (+ 1260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:33.624492 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:33.624515 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:33.624635 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:33.624641 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.626073 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.626076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:33.627893 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:33.627905 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.629361 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.629364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:33.629482 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:33.629509 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 18:36:33.629565 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 18:36:33.629672 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":213,"thread_start_us":161,"threads_started":1}
I20260929 18:36:33.630429 17243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:33.623605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45590 (local address 127.28.207.65:33073)
0929 18:36:33.623880 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:33.623887 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:33.623914 (+ 27us) server_negotiation.cc:407] Connection header received
0929 18:36:33.623995 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:33.624000 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:33.624222 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:33.624745 (+ 523us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:33.624906 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.626392 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.627968 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.628764 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.629861 (+ 1097us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:33.630101 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:33.630166 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 18:36:33.630268 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":93,"threads_started":1}
I20260929 18:36:33.630672 17244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:33.623628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39172 (local address 127.28.207.66:34033)
0929 18:36:33.624071 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:33.624082 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:33.624105 (+ 23us) server_negotiation.cc:407] Connection header received
0929 18:36:33.624176 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:33.624180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:33.624341 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:33.624496 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:33.624770 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.625902 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.628068 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.629197 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.629953 (+ 756us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:33.630175 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:33.630341 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 18:36:33.630476 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":233,"threads_started":1}
I20260929 18:36:33.630672 17240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:33.622234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33654 (local address 127.28.207.67:45971)
0929 18:36:33.622941 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:33.622957 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:33.622988 (+ 31us) server_negotiation.cc:407] Connection header received
0929 18:36:33.623605 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:33.623613 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:33.624006 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:33.624250 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:33.624617 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.625709 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.628054 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.629013 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.629142 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:33.629418 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:33.630330 (+ 912us) server_negotiation.cc:299] Negotiation successful
0929 18:36:33.630488 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":264,"threads_started":1}
I20260929 18:36:33.633764 17031 tablet_service.cc:1467] Processing CreateTablet for tablet 3052ba90001841cc87ad1635577413a7 (DEFAULT_TABLE table=test-table [id=c7641c33e7254ac88921c3ec8a282122]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:33.633805 16900 tablet_service.cc:1467] Processing CreateTablet for tablet 3052ba90001841cc87ad1635577413a7 (DEFAULT_TABLE table=test-table [id=c7641c33e7254ac88921c3ec8a282122]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:33.634927 17031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3052ba90001841cc87ad1635577413a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:33.635234 16900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3052ba90001841cc87ad1635577413a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:33.638010 17162 tablet_service.cc:1467] Processing CreateTablet for tablet 3052ba90001841cc87ad1635577413a7 (DEFAULT_TABLE table=test-table [id=c7641c33e7254ac88921c3ec8a282122]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 18:36:33.640471 17162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3052ba90001841cc87ad1635577413a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:33.641747 17245 tablet_bootstrap.cc:492] T 3052ba90001841cc87ad1635577413a7 P d058fe2e059b4412a7242e7ac6ebc571: Bootstrap starting.
I20260929 18:36:33.644642 17245 tablet_bootstrap.cc:654] T 3052ba90001841cc87ad1635577413a7 P d058fe2e059b4412a7242e7ac6ebc571: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:33.645442 17245 log.cc:824] T 3052ba90001841cc87ad1635577413a7 P d058fe2e059b4412a7242e7ac6ebc571: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:33.646255 17246 tablet_bootstrap.cc:492] T 3052ba90001841cc87ad1635577413a7 P 6cb485f27cfd4426b66edc0db974ebb8: Bootstrap starting.
I20260929 18:36:33.647851 17245 tablet_bootstrap.cc:492] T 3052ba90001841cc87ad1635577413a7 P d058fe2e059b4412a7242e7ac6ebc571: No bootstrap required, opened a new log
I20260929 18:36:33.648074 17245 ts_tablet_manager.cc:1397] T 3052ba90001841cc87ad1635577413a7 P d058fe2e059b4412a7242e7ac6ebc571: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260929 18:36:33.649205 17246 tablet_bootstrap.cc:654] T 3052ba90001841cc87ad1635577413a7 P 6cb485f27cfd4426b66edc0db974ebb8: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:33.649216 17248 tablet_bootstrap.cc:492] T 3052ba90001841cc87ad1635577413a7 P 25dca257cb1f43b5b46922156f2b7cf9: Bootstrap starting.
I20260929 18:36:33.650067 17246 log.cc:824] T 3052ba90001841cc87ad1635577413a7 P 6cb485f27cfd4426b66edc0db974ebb8: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:33.652226 17248 tablet_bootstrap.cc:654] T 3052ba90001841cc87ad1635577413a7 P 25dca257cb1f43b5b46922156f2b7cf9: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:33.652745 17245 raft_consensus.cc:348] T 3052ba90001841cc87ad1635577413a7 P d058fe2e059b4412a7242e7ac6ebc571 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25dca257cb1f43b5b46922156f2b7cf9" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 45971 } } peers { permanent_uuid: "d058fe2e059b4412a7242e7ac6ebc571" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 34033 } } peers { permanent_uuid: "6cb485f27cfd4426b66edc0db974ebb8" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 33073 } }
I20260929 18:36:33.653000 17245 raft_consensus.cc:374] T 3052ba90001841cc87ad1635577413a7 P d058fe2e059b4412a7242e7ac6ebc571 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:33.653061 17245 raft_consensus.cc:729] T 3052ba90001841cc87ad1635577413a7 P d058fe2e059b4412a7242e7ac6ebc571 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d058fe2e059b4412a7242e7ac6ebc571, State: Initialized, Role: FOLLOWER
I20260929 18:36:33.653370 17246 tablet_bootstrap.cc:492] T 3052ba90001841cc87ad1635577413a7 P 6cb485f27cfd4426b66edc0db974ebb8: No bootstrap required, opened a new log
I20260929 18:36:33.653518 17248 log.cc:824] T 3052ba90001841cc87ad1635577413a7 P 25dca257cb1f43b5b46922156f2b7cf9: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:33.653554 17245 consensus_queue.cc:260] T 3052ba90001841cc87ad1635577413a7 P d058fe2e059b4412a7242e7ac6ebc571 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25dca257cb1f43b5b46922156f2b7cf9" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 45971 } } peers { permanent_uuid: "d058fe2e059b4412a7242e7ac6ebc571" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 34033 } } peers { permanent_uuid: "6cb485f27cfd4426b66edc0db974ebb8" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 33073 } }
I20260929 18:36:33.653656 17246 ts_tablet_manager.cc:1397] T 3052ba90001841cc87ad1635577413a7 P 6cb485f27cfd4426b66edc0db974ebb8: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260929 18:36:33.654395 17097 heartbeater.cc:503] Master 127.28.207.126:42641 was elected leader, sending a full tablet report...
I20260929 18:36:33.654760 17245 ts_tablet_manager.cc:1428] T 3052ba90001841cc87ad1635577413a7 P d058fe2e059b4412a7242e7ac6ebc571: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260929 18:36:33.656366 17248 tablet_bootstrap.cc:492] T 3052ba90001841cc87ad1635577413a7 P 25dca257cb1f43b5b46922156f2b7cf9: No bootstrap required, opened a new log
I20260929 18:36:33.656607 17248 ts_tablet_manager.cc:1397] T 3052ba90001841cc87ad1635577413a7 P 25dca257cb1f43b5b46922156f2b7cf9: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 18:36:33.658480 17246 raft_consensus.cc:348] T 3052ba90001841cc87ad1635577413a7 P 6cb485f27cfd4426b66edc0db974ebb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25dca257cb1f43b5b46922156f2b7cf9" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 45971 } } peers { permanent_uuid: "d058fe2e059b4412a7242e7ac6ebc571" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 34033 } } peers { permanent_uuid: "6cb485f27cfd4426b66edc0db974ebb8" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 33073 } }
I20260929 18:36:33.659703 17246 raft_consensus.cc:374] T 3052ba90001841cc87ad1635577413a7 P 6cb485f27cfd4426b66edc0db974ebb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:33.659785 17246 raft_consensus.cc:729] T 3052ba90001841cc87ad1635577413a7 P 6cb485f27cfd4426b66edc0db974ebb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cb485f27cfd4426b66edc0db974ebb8, State: Initialized, Role: FOLLOWER
I20260929 18:36:33.660386 17246 consensus_queue.cc:260] T 3052ba90001841cc87ad1635577413a7 P 6cb485f27cfd4426b66edc0db974ebb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25dca257cb1f43b5b46922156f2b7cf9" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 45971 } } peers { permanent_uuid: "d058fe2e059b4412a7242e7ac6ebc571" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 34033 } } peers { permanent_uuid: "6cb485f27cfd4426b66edc0db974ebb8" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 33073 } }
I20260929 18:36:33.661015 17248 raft_consensus.cc:348] T 3052ba90001841cc87ad1635577413a7 P 25dca257cb1f43b5b46922156f2b7cf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25dca257cb1f43b5b46922156f2b7cf9" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 45971 } } peers { permanent_uuid: "d058fe2e059b4412a7242e7ac6ebc571" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 34033 } } peers { permanent_uuid: "6cb485f27cfd4426b66edc0db974ebb8" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 33073 } }
I20260929 18:36:33.661281 17248 raft_consensus.cc:374] T 3052ba90001841cc87ad1635577413a7 P 25dca257cb1f43b5b46922156f2b7cf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:33.661336 17248 raft_consensus.cc:729] T 3052ba90001841cc87ad1635577413a7 P 25dca257cb1f43b5b46922156f2b7cf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25dca257cb1f43b5b46922156f2b7cf9, State: Initialized, Role: FOLLOWER
I20260929 18:36:33.661466 16966 heartbeater.cc:503] Master 127.28.207.126:42641 was elected leader, sending a full tablet report...
I20260929 18:36:33.661767 17246 ts_tablet_manager.cc:1428] T 3052ba90001841cc87ad1635577413a7 P 6cb485f27cfd4426b66edc0db974ebb8: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260929 18:36:33.661801 17248 consensus_queue.cc:260] T 3052ba90001841cc87ad1635577413a7 P 25dca257cb1f43b5b46922156f2b7cf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25dca257cb1f43b5b46922156f2b7cf9" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 45971 } } peers { permanent_uuid: "d058fe2e059b4412a7242e7ac6ebc571" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 34033 } } peers { permanent_uuid: "6cb485f27cfd4426b66edc0db974ebb8" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 33073 } }
I20260929 18:36:33.662684 17228 heartbeater.cc:503] Master 127.28.207.126:42641 was elected leader, sending a full tablet report...
I20260929 18:36:33.662890 17248 ts_tablet_manager.cc:1428] T 3052ba90001841cc87ad1635577413a7 P 25dca257cb1f43b5b46922156f2b7cf9: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260929 18:36:33.669574 17250 raft_consensus.cc:482] T 3052ba90001841cc87ad1635577413a7 P d058fe2e059b4412a7242e7ac6ebc571 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 18:36:33.669860 17250 raft_consensus.cc:504] T 3052ba90001841cc87ad1635577413a7 P d058fe2e059b4412a7242e7ac6ebc571 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25dca257cb1f43b5b46922156f2b7cf9" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 45971 } } peers { permanent_uuid: "d058fe2e059b4412a7242e7ac6ebc571" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 34033 } } peers { permanent_uuid: "6cb485f27cfd4426b66edc0db974ebb8" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 33073 } }
I20260929 18:36:33.671753 17250 leader_election.cc:288] T 3052ba90001841cc87ad1635577413a7 P d058fe2e059b4412a7242e7ac6ebc571 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25dca257cb1f43b5b46922156f2b7cf9 (127.28.207.67:45971), 6cb485f27cfd4426b66edc0db974ebb8 (127.28.207.65:33073)
I20260929 18:36:33.676367 17254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:33.672054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:33073 (local address 127.28.207.66:54631)
0929 18:36:33.672591 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:33.672608 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:33.672749 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:33.673036 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:33.673039 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:33.673054 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:33.673220 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:33.673227 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.674028 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.674030 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:33.674995 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:33.675005 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.675840 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.675844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:33.675959 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:33.675992 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 18:36:33.676089 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 18:36:33.676176 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"mutex_wait_us":48,"thread_start_us":131,"threads_started":1}
I20260929 18:36:33.676397 17243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:33.672174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:54631 (local address 127.28.207.65:33073)
0929 18:36:33.672317 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:33.672322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:33.672793 (+ 471us) server_negotiation.cc:407] Connection header received
0929 18:36:33.672859 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:33.672861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:33.672918 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:33.673018 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:33.673342 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.673908 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.675133 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.675599 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.675717 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:33.675758 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:33.676156 (+ 398us) server_negotiation.cc:299] Negotiation successful
0929 18:36:33.676245 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:36:33.676664 17099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:33.672054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.67:45971 (local address 127.28.207.66:58479)
0929 18:36:33.672304 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:33.672320 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:36:33.672453 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:36:33.672830 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:36:33.672834 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:36:33.672863 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 18:36:33.673020 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:33.673029 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.673942 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.673949 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:33.675146 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:36:33.675156 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.676182 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.676185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:36:33.676294 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:33.676323 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 18:36:33.676421 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 18:36:33.676513 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":72}
I20260929 18:36:33.676713 17240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:33.672173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.28.207.66:58479 (local address 127.28.207.67:45971)
0929 18:36:33.672317 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:33.672322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:33.672398 (+ 76us) server_negotiation.cc:407] Connection header received
0929 18:36:33.672590 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:33.672593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:33.672670 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:33.672764 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 18:36:33.673188 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.673815 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.675378 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.676013 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.676138 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:33.676190 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:33.676461 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 18:36:33.676553 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 18:36:33.677420 16920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3052ba90001841cc87ad1635577413a7" candidate_uuid: "d058fe2e059b4412a7242e7ac6ebc571" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cb485f27cfd4426b66edc0db974ebb8" is_pre_election: true
I20260929 18:36:33.677551 17182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3052ba90001841cc87ad1635577413a7" candidate_uuid: "d058fe2e059b4412a7242e7ac6ebc571" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25dca257cb1f43b5b46922156f2b7cf9" is_pre_election: true
I20260929 18:36:33.677798 16920 raft_consensus.cc:2454] T 3052ba90001841cc87ad1635577413a7 P 6cb485f27cfd4426b66edc0db974ebb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d058fe2e059b4412a7242e7ac6ebc571 in term 0.
I20260929 18:36:33.678048 17182 raft_consensus.cc:2454] T 3052ba90001841cc87ad1635577413a7 P 25dca257cb1f43b5b46922156f2b7cf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d058fe2e059b4412a7242e7ac6ebc571 in term 0.
I20260929 18:36:33.678329 16987 leader_election.cc:302] T 3052ba90001841cc87ad1635577413a7 P d058fe2e059b4412a7242e7ac6ebc571 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cb485f27cfd4426b66edc0db974ebb8, d058fe2e059b4412a7242e7ac6ebc571; no voters:
I20260929 18:36:33.678709 17250 raft_consensus.cc:2781] T 3052ba90001841cc87ad1635577413a7 P d058fe2e059b4412a7242e7ac6ebc571 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 18:36:33.678818 17250 raft_consensus.cc:482] T 3052ba90001841cc87ad1635577413a7 P d058fe2e059b4412a7242e7ac6ebc571 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 18:36:33.678917 17250 raft_consensus.cc:3037] T 3052ba90001841cc87ad1635577413a7 P d058fe2e059b4412a7242e7ac6ebc571 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:33.680222 17250 raft_consensus.cc:504] T 3052ba90001841cc87ad1635577413a7 P d058fe2e059b4412a7242e7ac6ebc571 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25dca257cb1f43b5b46922156f2b7cf9" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 45971 } } peers { permanent_uuid: "d058fe2e059b4412a7242e7ac6ebc571" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 34033 } } peers { permanent_uuid: "6cb485f27cfd4426b66edc0db974ebb8" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 33073 } }
I20260929 18:36:33.680729 17250 leader_election.cc:288] T 3052ba90001841cc87ad1635577413a7 P d058fe2e059b4412a7242e7ac6ebc571 [CANDIDATE]: Term 1 election: Requested vote from peers 25dca257cb1f43b5b46922156f2b7cf9 (127.28.207.67:45971), 6cb485f27cfd4426b66edc0db974ebb8 (127.28.207.65:33073)
I20260929 18:36:33.681161 17182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3052ba90001841cc87ad1635577413a7" candidate_uuid: "d058fe2e059b4412a7242e7ac6ebc571" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25dca257cb1f43b5b46922156f2b7cf9"
I20260929 18:36:33.681217 16920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3052ba90001841cc87ad1635577413a7" candidate_uuid: "d058fe2e059b4412a7242e7ac6ebc571" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cb485f27cfd4426b66edc0db974ebb8"
I20260929 18:36:33.681335 17182 raft_consensus.cc:3037] T 3052ba90001841cc87ad1635577413a7 P 25dca257cb1f43b5b46922156f2b7cf9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:33.681382 16920 raft_consensus.cc:3037] T 3052ba90001841cc87ad1635577413a7 P 6cb485f27cfd4426b66edc0db974ebb8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:33.682587 17182 raft_consensus.cc:2454] T 3052ba90001841cc87ad1635577413a7 P 25dca257cb1f43b5b46922156f2b7cf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d058fe2e059b4412a7242e7ac6ebc571 in term 1.
I20260929 18:36:33.682760 16920 raft_consensus.cc:2454] T 3052ba90001841cc87ad1635577413a7 P 6cb485f27cfd4426b66edc0db974ebb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d058fe2e059b4412a7242e7ac6ebc571 in term 1.
I20260929 18:36:33.683183 16986 leader_election.cc:302] T 3052ba90001841cc87ad1635577413a7 P d058fe2e059b4412a7242e7ac6ebc571 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25dca257cb1f43b5b46922156f2b7cf9, d058fe2e059b4412a7242e7ac6ebc571; no voters:
I20260929 18:36:33.683398 17250 raft_consensus.cc:2781] T 3052ba90001841cc87ad1635577413a7 P d058fe2e059b4412a7242e7ac6ebc571 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:33.683656 17250 raft_consensus.cc:686] T 3052ba90001841cc87ad1635577413a7 P d058fe2e059b4412a7242e7ac6ebc571 [term 1 LEADER]: Becoming Leader. State: Replica: d058fe2e059b4412a7242e7ac6ebc571, State: Running, Role: LEADER
I20260929 18:36:33.684036 17250 consensus_queue.cc:237] T 3052ba90001841cc87ad1635577413a7 P d058fe2e059b4412a7242e7ac6ebc571 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25dca257cb1f43b5b46922156f2b7cf9" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 45971 } } peers { permanent_uuid: "d058fe2e059b4412a7242e7ac6ebc571" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 34033 } } peers { permanent_uuid: "6cb485f27cfd4426b66edc0db974ebb8" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 33073 } }
I20260929 18:36:33.687891 16778 catalog_manager.cc:5495] T 3052ba90001841cc87ad1635577413a7 P d058fe2e059b4412a7242e7ac6ebc571 reported cstate change: term changed from 0 to 1, leader changed from <none> to d058fe2e059b4412a7242e7ac6ebc571 (127.28.207.66). New cstate: current_term: 1 leader_uuid: "d058fe2e059b4412a7242e7ac6ebc571" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25dca257cb1f43b5b46922156f2b7cf9" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 45971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d058fe2e059b4412a7242e7ac6ebc571" member_type: VOTER last_known_addr { host: "127.28.207.66" port: 34033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6cb485f27cfd4426b66edc0db974ebb8" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 33073 } health_report { overall_health: UNKNOWN } } }
I20260929 18:36:33.710718 17244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:33.706513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39188 (local address 127.28.207.66:34033)
0929 18:36:33.706671 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:33.706681 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:33.706698 (+ 17us) server_negotiation.cc:407] Connection header received
0929 18:36:33.706783 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:33.706785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:33.706844 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:33.707007 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:36:33.707476 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.708029 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.709063 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:33.709309 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:33.709430 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:33.709753 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:36:33.710240 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:36:33.710344 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:33.710502 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 18:36:33.710569 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 18:36:33.718595 16920 raft_consensus.cc:1261] T 3052ba90001841cc87ad1635577413a7 P 6cb485f27cfd4426b66edc0db974ebb8 [term 1 FOLLOWER]: Refusing update from remote peer d058fe2e059b4412a7242e7ac6ebc571: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:36:33.718607 17182 raft_consensus.cc:1261] T 3052ba90001841cc87ad1635577413a7 P 25dca257cb1f43b5b46922156f2b7cf9 [term 1 FOLLOWER]: Refusing update from remote peer d058fe2e059b4412a7242e7ac6ebc571: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:36:33.719863 17255 consensus_queue.cc:1035] T 3052ba90001841cc87ad1635577413a7 P d058fe2e059b4412a7242e7ac6ebc571 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25dca257cb1f43b5b46922156f2b7cf9" member_type: VOTER last_known_addr { host: "127.28.207.67" port: 45971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:36:33.720144 17250 consensus_queue.cc:1035] T 3052ba90001841cc87ad1635577413a7 P d058fe2e059b4412a7242e7ac6ebc571 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cb485f27cfd4426b66edc0db974ebb8" member_type: VOTER last_known_addr { host: "127.28.207.65" port: 33073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:36:33.738471 16778 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:36:33.738662 16778 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 18:36:33.741873 16778 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0dd30edc9a9e4cfd9b58787191a6024a: Sending DeleteTablet for 3 replicas of tablet 3052ba90001841cc87ad1635577413a7
I20260929 18:36:33.743270 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 16838
I20260929 18:36:33.743561 17162 tablet_service.cc:1514] Processing DeleteTablet for tablet 3052ba90001841cc87ad1635577413a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:33 UTC) from {username='slave'} at 127.0.0.1:33654
I20260929 18:36:33.746112 17268 tablet_replica.cc:330] T 3052ba90001841cc87ad1635577413a7 P 25dca257cb1f43b5b46922156f2b7cf9: stopping tablet replica
I20260929 18:36:33.746126 17031 tablet_service.cc:1514] Processing DeleteTablet for tablet 3052ba90001841cc87ad1635577413a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 18:36:33 UTC) from {username='slave'} at 127.0.0.1:39172
I20260929 18:36:33.746591 17268 raft_consensus.cc:2229] T 3052ba90001841cc87ad1635577413a7 P 25dca257cb1f43b5b46922156f2b7cf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 18:36:33.747098 17268 raft_consensus.cc:2258] T 3052ba90001841cc87ad1635577413a7 P 25dca257cb1f43b5b46922156f2b7cf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:36:33.747105 17269 tablet_replica.cc:330] T 3052ba90001841cc87ad1635577413a7 P d058fe2e059b4412a7242e7ac6ebc571: stopping tablet replica
I20260929 18:36:33.747788 17269 raft_consensus.cc:2229] T 3052ba90001841cc87ad1635577413a7 P d058fe2e059b4412a7242e7ac6ebc571 [term 1 LEADER]: Raft consensus shutting down.
I20260929 18:36:33.748378 17269 raft_consensus.cc:2258] T 3052ba90001841cc87ad1635577413a7 P d058fe2e059b4412a7242e7ac6ebc571 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 18:36:33.749578 17268 ts_tablet_manager.cc:1905] T 3052ba90001841cc87ad1635577413a7 P 25dca257cb1f43b5b46922156f2b7cf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 18:36:33.750295 17269 ts_tablet_manager.cc:1905] T 3052ba90001841cc87ad1635577413a7 P d058fe2e059b4412a7242e7ac6ebc571: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 18:36:33.751932 16765 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.28.207.65:33073 (error 108)
I20260929 18:36:33.753448 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 16969
I20260929 18:36:33.754362 17239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:33.753162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.65:33073 (local address 127.0.0.1:45606)
0929 18:36:33.754110 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:33.754237 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.65:33073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":799}
W20260929 18:36:33.755712 16765 catalog_manager.cc:4584] TS 6cb485f27cfd4426b66edc0db974ebb8 (127.28.207.65:33073): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3052ba90001841cc87ad1635577413a7: Network error: Client connection negotiation failed: client connection to 127.28.207.65:33073: connect: Connection refused (error 111)
I20260929 18:36:33.756846 17268 ts_tablet_manager.cc:1918] T 3052ba90001841cc87ad1635577413a7 P 25dca257cb1f43b5b46922156f2b7cf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 18:36:33.756960 17268 log.cc:1196] T 3052ba90001841cc87ad1635577413a7 P 25dca257cb1f43b5b46922156f2b7cf9: Deleting WAL directory at /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790706718260899-29501-0/minicluster-data/ts-2/wal/wals/3052ba90001841cc87ad1635577413a7
I20260929 18:36:33.757296 17268 ts_tablet_manager.cc:1939] T 3052ba90001841cc87ad1635577413a7 P 25dca257cb1f43b5b46922156f2b7cf9: Deleting consensus metadata
I20260929 18:36:33.759014 16764 catalog_manager.cc:4841] TS 25dca257cb1f43b5b46922156f2b7cf9 (127.28.207.67:45971): tablet 3052ba90001841cc87ad1635577413a7 (table test-table [id=c7641c33e7254ac88921c3ec8a282122]) successfully deleted
I20260929 18:36:33.760373 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 17100
I20260929 18:36:33.761726 17239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:33.760637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.66:34033 (local address 127.0.0.1:39192)
0929 18:36:33.761520 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0929 18:36:33.761599 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.66:34033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":757}
I20260929 18:36:33.767470 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 16747
2026-09-29T18:36:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1228 ms)
[----------] 6 tests from EncryptionPolicyTest (11564 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:36:33 dist-test-slave-qcpv krb5kdc[17273](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:36:33 dist-test-slave-qcpv krb5kdc[17273](info): set up 2 sockets
Sep 29 18:36:33 dist-test-slave-qcpv krb5kdc[17273](info): commencing operation
krb5kdc: starting...
W20260929 18:36:34.847400 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:36:34 dist-test-slave-qcpv krb5kdc[17273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:36:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:36:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:36:35.003899 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43631
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:35.123183 17286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:35.123495 17286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:35.123540 17286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:35.127780 17286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:36:35.127868 17286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:35.127890 17286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:35.127907 17286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:36:35.127923 17286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:36:35.133745 17286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43631
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:35.134977 17286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:35.136061 17286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:35.146157 17294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:35.146248 17286 server_base.cc:1034] running on GCE node
W20260929 18:36:35.146199 17291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:35.146200 17292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:35.147125 17286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:35.148348 17286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:35.149578 17286 hybrid_clock.cc:648] HybridClock initialized: now 1790706995149561 us; error 36 us; skew 500 ppm
Sep 29 18:36:35 dist-test-slave-qcpv krb5kdc[17273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:35.153019 17286 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:36:35.154157 17286 webserver.cc:460] Webserver started at http://127.0.0.1:41059/ using document root <none> and password file <none>
I20260929 18:36:35.154763 17286 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:35.154853 17286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:35.155139 17286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:35.157356 17286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "53514c312b2f4f3cbca5770f1b78f664"
format_stamp: "Formatted at 2026-09-29 18:36:35 on dist-test-slave-qcpv"
server_key: "63d751a031e44137060e4c86082dcaff"
server_key_iv: "3fe4ce7d7f202399a6a8bdf959d6921d"
server_key_version: "encryptionkey@0"
I20260929 18:36:35.158043 17286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "53514c312b2f4f3cbca5770f1b78f664"
format_stamp: "Formatted at 2026-09-29 18:36:35 on dist-test-slave-qcpv"
server_key: "63d751a031e44137060e4c86082dcaff"
server_key_iv: "3fe4ce7d7f202399a6a8bdf959d6921d"
server_key_version: "encryptionkey@0"
I20260929 18:36:35.162254 17286 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 18:36:35.165273 17301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:35.167078 17286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:36:35.167263 17286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "53514c312b2f4f3cbca5770f1b78f664"
format_stamp: "Formatted at 2026-09-29 18:36:35 on dist-test-slave-qcpv"
server_key: "63d751a031e44137060e4c86082dcaff"
server_key_iv: "3fe4ce7d7f202399a6a8bdf959d6921d"
server_key_version: "encryptionkey@0"
I20260929 18:36:35.167403 17286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:35.183452 17286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:35.186818 17286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:35.187053 17286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:35.195369 17286 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43631
I20260929 18:36:35.195531 17353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43631 every 8 connection(s)
I20260929 18:36:35.196437 17286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:36:35.200152 17354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:35.203840 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 17286
I20260929 18:36:35.204016 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:36:35.204423 29501 external_mini_cluster.cc:1428] Setting key 49fd7b8a1bce6b1d2c2466ac2207e0d5
I20260929 18:36:35.208431 17354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53514c312b2f4f3cbca5770f1b78f664: Bootstrap starting.
I20260929 18:36:35.211874 17354 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53514c312b2f4f3cbca5770f1b78f664: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:35.212863 17354 log.cc:824] T 00000000000000000000000000000000 P 53514c312b2f4f3cbca5770f1b78f664: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:36:35 dist-test-slave-qcpv krb5kdc[17273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 18:36:35.215675 17354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53514c312b2f4f3cbca5770f1b78f664: No bootstrap required, opened a new log
I20260929 18:36:35.219336 17354 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53514c312b2f4f3cbca5770f1b78f664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53514c312b2f4f3cbca5770f1b78f664" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43631 } }
I20260929 18:36:35.219666 17354 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53514c312b2f4f3cbca5770f1b78f664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:35.219767 17354 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53514c312b2f4f3cbca5770f1b78f664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53514c312b2f4f3cbca5770f1b78f664, State: Initialized, Role: FOLLOWER
I20260929 18:36:35.220359 17354 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53514c312b2f4f3cbca5770f1b78f664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53514c312b2f4f3cbca5770f1b78f664" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43631 } }
I20260929 18:36:35.220531 17354 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53514c312b2f4f3cbca5770f1b78f664 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:36:35.220602 17354 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53514c312b2f4f3cbca5770f1b78f664 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:36:35.220712 17354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53514c312b2f4f3cbca5770f1b78f664 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:35.221705 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:35.206368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42657 (local address 127.0.0.1:43631)
0929 18:36:35.207012 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:35.207030 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:35.207072 (+ 42us) server_negotiation.cc:407] Connection header received
0929 18:36:35.208014 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:35.208036 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:35.208443 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:35.208857 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:35.209691 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:35.210739 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:35.211612 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:35.212162 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:35.212348 (+ 186us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:35.215865 (+ 3517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:35.215873 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:35.215895 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:35.215938 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:35.218295 (+ 2357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:35.218928 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:35.218936 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:35.218949 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:35.219075 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:35.219425 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:35.219435 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:35.219436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:35.219988 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:35.220310 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:35.220586 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 18:36:35.220856 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":192,"threads_started":1}
I20260929 18:36:35.221992 17354 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53514c312b2f4f3cbca5770f1b78f664 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53514c312b2f4f3cbca5770f1b78f664" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43631 } }
I20260929 18:36:35.222416 17354 leader_election.cc:302] T 00000000000000000000000000000000 P 53514c312b2f4f3cbca5770f1b78f664 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53514c312b2f4f3cbca5770f1b78f664; no voters:
I20260929 18:36:35.222841 17354 leader_election.cc:288] T 00000000000000000000000000000000 P 53514c312b2f4f3cbca5770f1b78f664 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:36:35.223065 17359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53514c312b2f4f3cbca5770f1b78f664 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:35.223351 17359 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53514c312b2f4f3cbca5770f1b78f664 [term 1 LEADER]: Becoming Leader. State: Replica: 53514c312b2f4f3cbca5770f1b78f664, State: Running, Role: LEADER
I20260929 18:36:35.223745 17359 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53514c312b2f4f3cbca5770f1b78f664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53514c312b2f4f3cbca5770f1b78f664" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43631 } }
I20260929 18:36:35.224474 17354 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53514c312b2f4f3cbca5770f1b78f664 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:36:35.225566 17359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53514c312b2f4f3cbca5770f1b78f664 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53514c312b2f4f3cbca5770f1b78f664. Latest consensus state: current_term: 1 leader_uuid: "53514c312b2f4f3cbca5770f1b78f664" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53514c312b2f4f3cbca5770f1b78f664" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43631 } } }
I20260929 18:36:35.225675 29501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:36:35.225711 17359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53514c312b2f4f3cbca5770f1b78f664 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:35.226192 17360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53514c312b2f4f3cbca5770f1b78f664 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53514c312b2f4f3cbca5770f1b78f664" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53514c312b2f4f3cbca5770f1b78f664" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43631 } } }
I20260929 18:36:35.226476 17360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53514c312b2f4f3cbca5770f1b78f664 [sys.catalog]: This master's current role is: LEADER
W20260929 18:36:35.231202 17374 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 53514c312b2f4f3cbca5770f1b78f664: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:36:35.231318 17374 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:36:35.231411 17368 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:36:35.232542 17368 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:36:35.239519 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:35.228335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40659 (local address 127.0.0.1:43631)
0929 18:36:35.228463 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:35.228468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:35.228717 (+ 249us) server_negotiation.cc:407] Connection header received
0929 18:36:35.228891 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:35.228897 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:35.228954 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:35.229031 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:35.230054 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:35.230992 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:35.232232 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:35.232522 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:35.232713 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:35.234680 (+ 1967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:35.234687 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:35.234691 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:35.234723 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:35.237023 (+ 2300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:35.237728 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:35.237732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:35.237735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:35.237853 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:35.238309 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:35.238312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:35.238314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:35.238588 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:35.238758 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:35.239166 (+ 408us) server_negotiation.cc:299] Negotiation successful
0929 18:36:35.239309 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:36:35.240017 17368 catalog_manager.cc:1347] Generated new cluster ID: fc7bd24d30764d128370367621a73018
I20260929 18:36:35.240593 17368 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:36:35.250425 17368 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:36:35.252903 17368 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:36:35.261307 17368 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53514c312b2f4f3cbca5770f1b78f664: Generated new TSK 0
I20260929 18:36:35.262264 17368 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:36:35.272082 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 17286
W20260929 18:36:35.279084 17271 connection.cc:535] client connection to 127.0.0.1:43631 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43631): Connection reset by peer (error 104)
2026-09-29T18:36:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1503 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:36:35 dist-test-slave-qcpv krb5kdc[17385](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:36:35 dist-test-slave-qcpv krb5kdc[17385](info): set up 2 sockets
Sep 29 18:36:35 dist-test-slave-qcpv krb5kdc[17385](info): commencing operation
krb5kdc: starting...
W20260929 18:36:36.331256 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:36:36 dist-test-slave-qcpv krb5kdc[17385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:36:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:36:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:36:36.474598 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39119
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:36.586092 17398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:36.586406 17398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:36.586450 17398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:36.590621 17398 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:36:36.590802 17398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:36.590957 17398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:36.590994 17398 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:36:36.591012 17398 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:36:36.597028 17398 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39119
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:36.598276 17398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:36.599390 17398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:36.609179 17406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:36.609179 17403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:36.609263 17404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:36.609277 17398 server_base.cc:1034] running on GCE node
I20260929 18:36:36.610196 17398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:36.611591 17398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:36.612922 17398 hybrid_clock.cc:648] HybridClock initialized: now 1790706996612876 us; error 78 us; skew 500 ppm
Sep 29 18:36:36 dist-test-slave-qcpv krb5kdc[17385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:36.616766 17398 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:36:36.617864 17398 webserver.cc:460] Webserver started at http://127.0.0.1:45459/ using document root <none> and password file <none>
I20260929 18:36:36.618476 17398 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:36.618577 17398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:36.618839 17398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:36.621088 17398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "946f6581e43f475994a360d7b077e62c"
format_stamp: "Formatted at 2026-09-29 18:36:36 on dist-test-slave-qcpv"
server_key: "308e5a4f9972663d3de3d31a8fc0d6d3"
server_key_iv: "0160fb4a65c95777b94eae6aed13153d"
server_key_version: "encryptionkey@0"
I20260929 18:36:36.621695 17398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "946f6581e43f475994a360d7b077e62c"
format_stamp: "Formatted at 2026-09-29 18:36:36 on dist-test-slave-qcpv"
server_key: "308e5a4f9972663d3de3d31a8fc0d6d3"
server_key_iv: "0160fb4a65c95777b94eae6aed13153d"
server_key_version: "encryptionkey@0"
I20260929 18:36:36.626194 17398 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:36:36.629159 17413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:36.630757 17398 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:36:36.630973 17398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "946f6581e43f475994a360d7b077e62c"
format_stamp: "Formatted at 2026-09-29 18:36:36 on dist-test-slave-qcpv"
server_key: "308e5a4f9972663d3de3d31a8fc0d6d3"
server_key_iv: "0160fb4a65c95777b94eae6aed13153d"
server_key_version: "encryptionkey@0"
I20260929 18:36:36.631105 17398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:36.660442 17398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:36.664539 17398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:36.664737 17398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:36.673923 17398 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39119
I20260929 18:36:36.674048 17465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39119 every 8 connection(s)
I20260929 18:36:36.675274 17398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:36:36.678777 17466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:36.684116 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 17398
I20260929 18:36:36.684297 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:36:36.684712 29501 external_mini_cluster.cc:1428] Setting key 1aa47065b3584c1717c9f930a5eafcf9
I20260929 18:36:36.685171 17466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 946f6581e43f475994a360d7b077e62c: Bootstrap starting.
I20260929 18:36:36.688046 17466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 946f6581e43f475994a360d7b077e62c: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:36.688938 17466 log.cc:824] T 00000000000000000000000000000000 P 946f6581e43f475994a360d7b077e62c: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:36.691308 17466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 946f6581e43f475994a360d7b077e62c: No bootstrap required, opened a new log
I20260929 18:36:36.696127 17466 raft_consensus.cc:348] T 00000000000000000000000000000000 P 946f6581e43f475994a360d7b077e62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "946f6581e43f475994a360d7b077e62c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39119 } }
I20260929 18:36:36.696413 17466 raft_consensus.cc:374] T 00000000000000000000000000000000 P 946f6581e43f475994a360d7b077e62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:36.696453 17466 raft_consensus.cc:729] T 00000000000000000000000000000000 P 946f6581e43f475994a360d7b077e62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 946f6581e43f475994a360d7b077e62c, State: Initialized, Role: FOLLOWER
I20260929 18:36:36.696640 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:36.686524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48999 (local address 127.0.0.1:39119)
0929 18:36:36.687099 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:36.687130 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:36.687176 (+ 46us) server_negotiation.cc:407] Connection header received
0929 18:36:36.687999 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:36.688028 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:36.688539 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:36.688933 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:36.689425 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:36.690506 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:36.691943 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:36.692523 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:36.692693 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:36.693859 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:36.693865 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:36.693882 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:36.694460 (+ 578us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:36.694549 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:36.694598 (+ 49us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:36.694726 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:36.695087 (+ 361us) server_negotiation.cc:299] Negotiation successful
0929 18:36:36.695284 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":211,"threads_started":1}
I20260929 18:36:36.697050 17466 consensus_queue.cc:260] T 00000000000000000000000000000000 P 946f6581e43f475994a360d7b077e62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "946f6581e43f475994a360d7b077e62c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39119 } }
I20260929 18:36:36.697237 17466 raft_consensus.cc:388] T 00000000000000000000000000000000 P 946f6581e43f475994a360d7b077e62c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:36:36.697319 17466 raft_consensus.cc:482] T 00000000000000000000000000000000 P 946f6581e43f475994a360d7b077e62c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:36:36.697432 17466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 946f6581e43f475994a360d7b077e62c [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:36.698580 17466 raft_consensus.cc:504] T 00000000000000000000000000000000 P 946f6581e43f475994a360d7b077e62c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "946f6581e43f475994a360d7b077e62c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39119 } }
I20260929 18:36:36.699038 17466 leader_election.cc:302] T 00000000000000000000000000000000 P 946f6581e43f475994a360d7b077e62c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 946f6581e43f475994a360d7b077e62c; no voters:
I20260929 18:36:36.699466 17466 leader_election.cc:288] T 00000000000000000000000000000000 P 946f6581e43f475994a360d7b077e62c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:36:36.699622 17471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 946f6581e43f475994a360d7b077e62c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:36.699877 17471 raft_consensus.cc:686] T 00000000000000000000000000000000 P 946f6581e43f475994a360d7b077e62c [term 1 LEADER]: Becoming Leader. State: Replica: 946f6581e43f475994a360d7b077e62c, State: Running, Role: LEADER
I20260929 18:36:36.700205 17471 consensus_queue.cc:237] T 00000000000000000000000000000000 P 946f6581e43f475994a360d7b077e62c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "946f6581e43f475994a360d7b077e62c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39119 } }
I20260929 18:36:36.700835 29501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:36:36.700961 17466 sys_catalog.cc:564] T 00000000000000000000000000000000 P 946f6581e43f475994a360d7b077e62c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:36:36.702052 17473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 946f6581e43f475994a360d7b077e62c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 946f6581e43f475994a360d7b077e62c. Latest consensus state: current_term: 1 leader_uuid: "946f6581e43f475994a360d7b077e62c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "946f6581e43f475994a360d7b077e62c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39119 } } }
I20260929 18:36:36.702044 17472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 946f6581e43f475994a360d7b077e62c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "946f6581e43f475994a360d7b077e62c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "946f6581e43f475994a360d7b077e62c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39119 } } }
I20260929 18:36:36.702190 17473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 946f6581e43f475994a360d7b077e62c [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:36.702359 17472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 946f6581e43f475994a360d7b077e62c [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:36.702956 17484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:36:36.706760 17484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:36:36.709478 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:36.703926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60231 (local address 127.0.0.1:39119)
0929 18:36:36.704078 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:36.704086 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:36.704482 (+ 396us) server_negotiation.cc:407] Connection header received
0929 18:36:36.704707 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:36.704714 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:36.704790 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:36.704852 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:36.705286 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:36.705886 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:36.707277 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:36.707734 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:36.707811 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:36.708916 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:36.708920 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:36.708923 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:36.708939 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:36.708969 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:36.708983 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:36.709107 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:36.709247 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 18:36:36.709297 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 18:36:36.714115 17484 catalog_manager.cc:1347] Generated new cluster ID: aa6a5ec3e3394ff4a359bf303a87830c
I20260929 18:36:36.714222 17484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:36:36.722548 17484 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:36:36.723731 17484 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:36:36.728156 17484 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 946f6581e43f475994a360d7b077e62c: Generated new TSK 0
I20260929 18:36:36.728935 17484 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:36:36.737028 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 17398
W20260929 18:36:36.743031 17383 connection.cc:535] client connection to 127.0.0.1:39119 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39119): Connection reset by peer (error 104)
2026-09-29T18:36:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1463 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:36:36 dist-test-slave-qcpv krb5kdc[17497](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:36:36 dist-test-slave-qcpv krb5kdc[17497](info): set up 2 sockets
Sep 29 18:36:36 dist-test-slave-qcpv krb5kdc[17497](info): commencing operation
krb5kdc: starting...
W20260929 18:36:37.799264 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:36:37 dist-test-slave-qcpv krb5kdc[17497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:36:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:36:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:36:37.949290 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46479
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:38.097760 17510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:38.098207 17510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:38.098299 17510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:38.102777 17510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:36:38.102931 17510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:38.102980 17510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:38.103049 17510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:36:38.103089 17510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:36:38.109635 17510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46479
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:38.111248 17510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:38.112600 17510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:38.123572 17518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:38.123589 17515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:38.123880 17510 server_base.cc:1034] running on GCE node
W20260929 18:36:38.123589 17516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:38.124713 17510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:38.126134 17510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:38.127487 17510 hybrid_clock.cc:648] HybridClock initialized: now 1790706998127435 us; error 69 us; skew 500 ppm
Sep 29 18:36:38 dist-test-slave-qcpv krb5kdc[17497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:38.139478 17510 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:36:38.141616 17510 webserver.cc:460] Webserver started at http://127.0.0.1:34885/ using document root <none> and password file <none>
I20260929 18:36:38.142683 17510 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:38.142802 17510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:38.143146 17510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:38.146152 17510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "65694e05a1d74147b6dc567fec5a6156"
format_stamp: "Formatted at 2026-09-29 18:36:38 on dist-test-slave-qcpv"
server_key: "4e9d67fe66cc349e7292d26306c7cfba"
server_key_iv: "898ed9c8554975dca00e548b07545112"
server_key_version: "encryptionkey@0"
I20260929 18:36:38.147212 17510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "65694e05a1d74147b6dc567fec5a6156"
format_stamp: "Formatted at 2026-09-29 18:36:38 on dist-test-slave-qcpv"
server_key: "4e9d67fe66cc349e7292d26306c7cfba"
server_key_iv: "898ed9c8554975dca00e548b07545112"
server_key_version: "encryptionkey@0"
I20260929 18:36:38.152468 17510 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 18:36:38.155625 17525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:38.157176 17510 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 18:36:38.157300 17510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "65694e05a1d74147b6dc567fec5a6156"
format_stamp: "Formatted at 2026-09-29 18:36:38 on dist-test-slave-qcpv"
server_key: "4e9d67fe66cc349e7292d26306c7cfba"
server_key_iv: "898ed9c8554975dca00e548b07545112"
server_key_version: "encryptionkey@0"
I20260929 18:36:38.157419 17510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:38.162346 17510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:38.165910 17510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:38.166081 17510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:38.173650 17510 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46479
I20260929 18:36:38.173681 17577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46479 every 8 connection(s)
I20260929 18:36:38.174661 17510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:36:38.178532 17578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:38.178577 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 17510
I20260929 18:36:38.178743 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:36:38.179109 29501 external_mini_cluster.cc:1428] Setting key 64b74dd44ce61eb458b8f8492cede590
I20260929 18:36:38.185724 17581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:38.180804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33479 (local address 127.0.0.1:46479)
0929 18:36:38.181402 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:38.181422 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:38.181470 (+ 48us) server_negotiation.cc:407] Connection header received
0929 18:36:38.182587 (+ 1117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:38.182609 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:38.183047 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:38.183434 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:38.184413 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:38.184420 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:38.184439 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:38.184451 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:38.184508 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:38.184529 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:38.184697 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:38.184768 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 18:36:38.184938 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":188,"threads_started":1}
I20260929 18:36:38.188089 17578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65694e05a1d74147b6dc567fec5a6156: Bootstrap starting.
I20260929 18:36:38.188882 29501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:36:38.191325 17578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65694e05a1d74147b6dc567fec5a6156: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:38.192649 17578 log.cc:824] T 00000000000000000000000000000000 P 65694e05a1d74147b6dc567fec5a6156: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:38.193248 17581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:38.191077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55773 (local address 127.0.0.1:46479)
0929 18:36:38.191194 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:38.191198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:38.191525 (+ 327us) server_negotiation.cc:407] Connection header received
0929 18:36:38.191679 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:38.191683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:38.191751 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:38.191851 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:38.192635 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:38.192639 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:38.192643 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:38.192669 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:38.192702 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:38.192718 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:38.192847 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:38.193044 (+ 197us) server_negotiation.cc:299] Negotiation successful
0929 18:36:38.193093 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 18:36:38.195580 17578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65694e05a1d74147b6dc567fec5a6156: No bootstrap required, opened a new log
I20260929 18:36:38.200241 17578 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65694e05a1d74147b6dc567fec5a6156 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65694e05a1d74147b6dc567fec5a6156" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46479 } }
I20260929 18:36:38.200733 17578 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65694e05a1d74147b6dc567fec5a6156 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:38.200875 17578 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65694e05a1d74147b6dc567fec5a6156 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65694e05a1d74147b6dc567fec5a6156, State: Initialized, Role: FOLLOWER
I20260929 18:36:38.201555 17578 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65694e05a1d74147b6dc567fec5a6156 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65694e05a1d74147b6dc567fec5a6156" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46479 } }
I20260929 18:36:38.201767 17578 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65694e05a1d74147b6dc567fec5a6156 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:36:38.201865 17578 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65694e05a1d74147b6dc567fec5a6156 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:36:38.202039 17578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65694e05a1d74147b6dc567fec5a6156 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:38.203418 17578 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65694e05a1d74147b6dc567fec5a6156 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65694e05a1d74147b6dc567fec5a6156" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46479 } }
I20260929 18:36:38.203867 17578 leader_election.cc:302] T 00000000000000000000000000000000 P 65694e05a1d74147b6dc567fec5a6156 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65694e05a1d74147b6dc567fec5a6156; no voters:
I20260929 18:36:38.204537 17578 leader_election.cc:288] T 00000000000000000000000000000000 P 65694e05a1d74147b6dc567fec5a6156 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:36:38.204666 17588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65694e05a1d74147b6dc567fec5a6156 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:38.204950 17588 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65694e05a1d74147b6dc567fec5a6156 [term 1 LEADER]: Becoming Leader. State: Replica: 65694e05a1d74147b6dc567fec5a6156, State: Running, Role: LEADER
I20260929 18:36:38.205344 17588 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65694e05a1d74147b6dc567fec5a6156 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65694e05a1d74147b6dc567fec5a6156" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46479 } }
I20260929 18:36:38.206588 17578 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65694e05a1d74147b6dc567fec5a6156 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:36:38.207345 17590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65694e05a1d74147b6dc567fec5a6156 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65694e05a1d74147b6dc567fec5a6156. Latest consensus state: current_term: 1 leader_uuid: "65694e05a1d74147b6dc567fec5a6156" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65694e05a1d74147b6dc567fec5a6156" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46479 } } }
I20260929 18:36:38.207521 17590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65694e05a1d74147b6dc567fec5a6156 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:38.207332 17589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65694e05a1d74147b6dc567fec5a6156 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65694e05a1d74147b6dc567fec5a6156" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65694e05a1d74147b6dc567fec5a6156" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46479 } } }
I20260929 18:36:38.207932 17589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65694e05a1d74147b6dc567fec5a6156 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:38.208012 17594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:36:38.211414 17594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:36:38.219357 17594 catalog_manager.cc:1347] Generated new cluster ID: a2f8673497f440f6b5e314be5ba6f2d0
I20260929 18:36:38.219471 17594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:36:38.245015 17594 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:36:38.245551 17594 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:36:38.254011 17594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65694e05a1d74147b6dc567fec5a6156: Generated new TSK 0
I20260929 18:36:38.255105 17594 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:36:38.268532 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 17510
2026-09-29T18:36:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1533 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:36:38 dist-test-slave-qcpv krb5kdc[17609](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:36:38 dist-test-slave-qcpv krb5kdc[17609](info): set up 2 sockets
Sep 29 18:36:38 dist-test-slave-qcpv krb5kdc[17609](info): commencing operation
krb5kdc: starting...
W20260929 18:36:39.340090 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:36:39 dist-test-slave-qcpv krb5kdc[17609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:36:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:36:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:36:39.486505 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45835
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:39.603128 17622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:39.603482 17622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:39.603569 17622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:39.607764 17622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:36:39.607952 17622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:39.608062 17622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:39.608129 17622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:36:39.608183 17622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:36:39.615939 17622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:39.617592 17622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:39.618956 17622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:39.629132 17628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:39.629165 17630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:39.629137 17627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:39.629583 17622 server_base.cc:1034] running on GCE node
I20260929 18:36:39.630403 17622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:39.631930 17622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:39.633172 17622 hybrid_clock.cc:648] HybridClock initialized: now 1790706999633128 us; error 60 us; skew 500 ppm
Sep 29 18:36:39 dist-test-slave-qcpv krb5kdc[17609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:39.636883 17622 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:36:39.638247 17622 webserver.cc:460] Webserver started at http://127.0.0.1:43651/ using document root <none> and password file <none>
I20260929 18:36:39.639160 17622 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:39.639271 17622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:39.639611 17622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:39.642159 17622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "5f0b9e8e4e80435483880687512d1403"
format_stamp: "Formatted at 2026-09-29 18:36:39 on dist-test-slave-qcpv"
server_key: "2acbf9ff856486a3b799329f5ddf00a9"
server_key_iv: "4d5fcaaa5db13b59863553ddb56b9924"
server_key_version: "encryptionkey@0"
I20260929 18:36:39.643052 17622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "5f0b9e8e4e80435483880687512d1403"
format_stamp: "Formatted at 2026-09-29 18:36:39 on dist-test-slave-qcpv"
server_key: "2acbf9ff856486a3b799329f5ddf00a9"
server_key_iv: "4d5fcaaa5db13b59863553ddb56b9924"
server_key_version: "encryptionkey@0"
I20260929 18:36:39.648361 17622 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 18:36:39.651860 17637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:39.653663 17622 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 18:36:39.653947 17622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "5f0b9e8e4e80435483880687512d1403"
format_stamp: "Formatted at 2026-09-29 18:36:39 on dist-test-slave-qcpv"
server_key: "2acbf9ff856486a3b799329f5ddf00a9"
server_key_iv: "4d5fcaaa5db13b59863553ddb56b9924"
server_key_version: "encryptionkey@0"
I20260929 18:36:39.654145 17622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:39.670763 17622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:39.675233 17622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:39.675664 17622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:39.686268 17622 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45835
I20260929 18:36:39.686290 17689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45835 every 8 connection(s)
I20260929 18:36:39.687386 17622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:36:39.690254 17690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:39.690666 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 17622
I20260929 18:36:39.690806 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:36:39.691164 29501 external_mini_cluster.cc:1428] Setting key 00e1d3d5af4eac899db318b577f52a83
I20260929 18:36:39.697454 17690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f0b9e8e4e80435483880687512d1403: Bootstrap starting.
I20260929 18:36:39.699806 17690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f0b9e8e4e80435483880687512d1403: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:39.700560 17690 log.cc:824] T 00000000000000000000000000000000 P 5f0b9e8e4e80435483880687512d1403: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:39.703276 17690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f0b9e8e4e80435483880687512d1403: No bootstrap required, opened a new log
Sep 29 18:36:39 dist-test-slave-qcpv krb5kdc[17609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790706999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 18:36:39.707834 17690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f0b9e8e4e80435483880687512d1403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f0b9e8e4e80435483880687512d1403" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45835 } }
I20260929 18:36:39.708140 17690 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f0b9e8e4e80435483880687512d1403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:39.708195 17690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f0b9e8e4e80435483880687512d1403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f0b9e8e4e80435483880687512d1403, State: Initialized, Role: FOLLOWER
I20260929 18:36:39.708688 17690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f0b9e8e4e80435483880687512d1403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f0b9e8e4e80435483880687512d1403" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45835 } }
I20260929 18:36:39.708822 17690 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f0b9e8e4e80435483880687512d1403 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:36:39.708917 17690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f0b9e8e4e80435483880687512d1403 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:36:39.709045 17690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f0b9e8e4e80435483880687512d1403 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:39.710150 17690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f0b9e8e4e80435483880687512d1403 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f0b9e8e4e80435483880687512d1403" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45835 } }
I20260929 18:36:39.710533 17690 leader_election.cc:302] T 00000000000000000000000000000000 P 5f0b9e8e4e80435483880687512d1403 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f0b9e8e4e80435483880687512d1403; no voters:
I20260929 18:36:39.710940 17690 leader_election.cc:288] T 00000000000000000000000000000000 P 5f0b9e8e4e80435483880687512d1403 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:36:39.711138 17695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f0b9e8e4e80435483880687512d1403 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:39.712561 17690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f0b9e8e4e80435483880687512d1403 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:36:39.713099 17695 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f0b9e8e4e80435483880687512d1403 [term 1 LEADER]: Becoming Leader. State: Replica: 5f0b9e8e4e80435483880687512d1403, State: Running, Role: LEADER
I20260929 18:36:39.713465 17695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f0b9e8e4e80435483880687512d1403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f0b9e8e4e80435483880687512d1403" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45835 } }
I20260929 18:36:39.713986 17693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:39.693008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32889 (local address 127.0.0.1:45835)
0929 18:36:39.693906 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:39.693943 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:39.694104 (+ 161us) server_negotiation.cc:407] Connection header received
0929 18:36:39.695720 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:39.695747 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:39.696388 (+ 641us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:39.696893 (+ 505us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:39.698579 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:39.700001 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:39.701297 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:39.702020 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:39.702232 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:39.705714 (+ 3482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:39.705721 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:39.705738 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:39.705771 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:39.708282 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:39.709173 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:39.709180 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:39.709187 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:39.709340 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:39.709959 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:39.709963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:39.709967 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:39.710524 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:39.710729 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:39.711950 (+ 1221us) server_negotiation.cc:299] Negotiation successful
0929 18:36:39.712484 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":142,"threads_started":1}
I20260929 18:36:39.716593 17697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0b9e8e4e80435483880687512d1403 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f0b9e8e4e80435483880687512d1403. Latest consensus state: current_term: 1 leader_uuid: "5f0b9e8e4e80435483880687512d1403" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f0b9e8e4e80435483880687512d1403" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45835 } } }
I20260929 18:36:39.716810 17697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0b9e8e4e80435483880687512d1403 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:39.717020 17696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0b9e8e4e80435483880687512d1403 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f0b9e8e4e80435483880687512d1403" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f0b9e8e4e80435483880687512d1403" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45835 } } }
I20260929 18:36:39.717190 17696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0b9e8e4e80435483880687512d1403 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:39.717985 29501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260929 18:36:39.720800 17710 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5f0b9e8e4e80435483880687512d1403: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:36:39.720954 17710 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:36:39.721041 17709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:36:39.721913 17709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:36:39.727782 17709 catalog_manager.cc:1347] Generated new cluster ID: b46a550310c444d7b0f70a6673331fd9
I20260929 18:36:39.727852 17709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:36:39.733942 17693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:39.720453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53353 (local address 127.0.0.1:45835)
0929 18:36:39.720576 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:39.720580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:39.721013 (+ 433us) server_negotiation.cc:407] Connection header received
0929 18:36:39.721278 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:39.721284 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:39.721337 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:39.721442 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:39.722412 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:39.723249 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:39.724997 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:39.725566 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:39.725694 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:39.729466 (+ 3772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:39.729471 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:39.729475 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:39.729515 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:39.731611 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:39.732225 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:39.732229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:39.732231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:39.732344 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:39.732696 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:39.732700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:39.732702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:39.732986 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:39.733110 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:39.733555 (+ 445us) server_negotiation.cc:299] Negotiation successful
0929 18:36:39.733723 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 18:36:39.742846 17709 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:36:39.744058 17709 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:36:39.749985 17709 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f0b9e8e4e80435483880687512d1403: Generated new TSK 0
I20260929 18:36:39.750936 17709 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:36:39.756219 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 17622
2026-09-29T18:36:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1486 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:36:39 dist-test-slave-qcpv krb5kdc[17721](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:36:39 dist-test-slave-qcpv krb5kdc[17721](info): set up 2 sockets
Sep 29 18:36:39 dist-test-slave-qcpv krb5kdc[17721](info): commencing operation
krb5kdc: starting...
W20260929 18:36:40.823331 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:36:40 dist-test-slave-qcpv krb5kdc[17721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:36:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:36:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:36:40.967532 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42709
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:41.123631 17734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:41.124197 17734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:41.124341 17734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:41.129194 17734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:36:41.129343 17734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:41.129380 17734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:41.129426 17734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:36:41.129462 17734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:36:41.135696 17734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:41.137594 17734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:41.139240 17734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:41.150803 17742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:41.150784 17739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:41.150784 17740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:41.151453 17734 server_base.cc:1034] running on GCE node
I20260929 18:36:41.152112 17734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:41.153411 17734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:41.154680 17734 hybrid_clock.cc:648] HybridClock initialized: now 1790707001154655 us; error 48 us; skew 500 ppm
Sep 29 18:36:41 dist-test-slave-qcpv krb5kdc[17721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707001, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:41.158399 17734 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:36:41.159555 17734 webserver.cc:460] Webserver started at http://127.0.0.1:42853/ using document root <none> and password file <none>
I20260929 18:36:41.160169 17734 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:41.160241 17734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:41.160427 17734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:41.162653 17734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "75bb6f7d5b8a448d98ebafeb6cc47b40"
format_stamp: "Formatted at 2026-09-29 18:36:41 on dist-test-slave-qcpv"
server_key: "16d7ae71e871f5d4e5df6af3c0aba83c"
server_key_iv: "acb7abb227045fa3ddfd399f0ca42dca"
server_key_version: "encryptionkey@0"
I20260929 18:36:41.163298 17734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "75bb6f7d5b8a448d98ebafeb6cc47b40"
format_stamp: "Formatted at 2026-09-29 18:36:41 on dist-test-slave-qcpv"
server_key: "16d7ae71e871f5d4e5df6af3c0aba83c"
server_key_iv: "acb7abb227045fa3ddfd399f0ca42dca"
server_key_version: "encryptionkey@0"
I20260929 18:36:41.167848 17734 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:36:41.170609 17749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:41.172159 17734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:36:41.172286 17734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "75bb6f7d5b8a448d98ebafeb6cc47b40"
format_stamp: "Formatted at 2026-09-29 18:36:41 on dist-test-slave-qcpv"
server_key: "16d7ae71e871f5d4e5df6af3c0aba83c"
server_key_iv: "acb7abb227045fa3ddfd399f0ca42dca"
server_key_version: "encryptionkey@0"
I20260929 18:36:41.172443 17734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:41.197250 17734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:41.201017 17734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:41.201210 17734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:41.210143 17734 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42709
I20260929 18:36:41.210333 17801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42709 every 8 connection(s)
I20260929 18:36:41.211385 17734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:36:41.215533 17802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:41.217751 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 17734
I20260929 18:36:41.217942 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:36:41.218333 29501 external_mini_cluster.cc:1428] Setting key 3cfd845bc25bdffecff540d9ea818216
I20260929 18:36:41.224713 17802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75bb6f7d5b8a448d98ebafeb6cc47b40: Bootstrap starting.
I20260929 18:36:41.228354 17802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75bb6f7d5b8a448d98ebafeb6cc47b40: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:41.229256 17802 log.cc:824] T 00000000000000000000000000000000 P 75bb6f7d5b8a448d98ebafeb6cc47b40: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:41.229362 17805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:41.220328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50929 (local address 127.0.0.1:42709)
0929 18:36:41.220815 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:41.220831 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:41.220872 (+ 41us) server_negotiation.cc:407] Connection header received
0929 18:36:41.221651 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:41.221670 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:41.222070 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:41.222395 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:41.222974 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:41.224046 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:41.225441 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:41.225985 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:41.226134 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:41.227245 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:41.227251 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:41.227265 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:41.227711 (+ 446us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:41.227773 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:41.227803 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:41.227895 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:41.228120 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 18:36:41.228388 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":182,"threads_started":1}
I20260929 18:36:41.232137 17802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75bb6f7d5b8a448d98ebafeb6cc47b40: No bootstrap required, opened a new log
I20260929 18:36:41.233394 29501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:36:41.237257 17802 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75bb6f7d5b8a448d98ebafeb6cc47b40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75bb6f7d5b8a448d98ebafeb6cc47b40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42709 } }
I20260929 18:36:41.237751 17802 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75bb6f7d5b8a448d98ebafeb6cc47b40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:41.237833 17802 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75bb6f7d5b8a448d98ebafeb6cc47b40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75bb6f7d5b8a448d98ebafeb6cc47b40, State: Initialized, Role: FOLLOWER
I20260929 18:36:41.238391 17802 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75bb6f7d5b8a448d98ebafeb6cc47b40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75bb6f7d5b8a448d98ebafeb6cc47b40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42709 } }
I20260929 18:36:41.238549 17802 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75bb6f7d5b8a448d98ebafeb6cc47b40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:36:41.238648 17802 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75bb6f7d5b8a448d98ebafeb6cc47b40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:36:41.238806 17802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75bb6f7d5b8a448d98ebafeb6cc47b40 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:41.240461 17802 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75bb6f7d5b8a448d98ebafeb6cc47b40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75bb6f7d5b8a448d98ebafeb6cc47b40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42709 } }
I20260929 18:36:41.240994 17802 leader_election.cc:302] T 00000000000000000000000000000000 P 75bb6f7d5b8a448d98ebafeb6cc47b40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75bb6f7d5b8a448d98ebafeb6cc47b40; no voters:
I20260929 18:36:41.241294 17805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:41.235748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37803 (local address 127.0.0.1:42709)
0929 18:36:41.235878 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:41.235881 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:41.236359 (+ 478us) server_negotiation.cc:407] Connection header received
0929 18:36:41.236547 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:41.236549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:41.236609 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:41.236698 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:41.237118 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:41.237674 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:41.239025 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:41.239299 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:41.239398 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:41.240539 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:41.240542 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:41.240544 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:41.240558 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:41.240588 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:41.240599 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:41.240725 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:41.240900 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 18:36:41.240963 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 18:36:41.241627 17802 leader_election.cc:288] T 00000000000000000000000000000000 P 75bb6f7d5b8a448d98ebafeb6cc47b40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:36:41.241776 17812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75bb6f7d5b8a448d98ebafeb6cc47b40 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:41.242113 17812 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75bb6f7d5b8a448d98ebafeb6cc47b40 [term 1 LEADER]: Becoming Leader. State: Replica: 75bb6f7d5b8a448d98ebafeb6cc47b40, State: Running, Role: LEADER
I20260929 18:36:41.242645 17812 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75bb6f7d5b8a448d98ebafeb6cc47b40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75bb6f7d5b8a448d98ebafeb6cc47b40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42709 } }
I20260929 18:36:41.243471 17802 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75bb6f7d5b8a448d98ebafeb6cc47b40 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:36:41.244853 17814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75bb6f7d5b8a448d98ebafeb6cc47b40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75bb6f7d5b8a448d98ebafeb6cc47b40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75bb6f7d5b8a448d98ebafeb6cc47b40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42709 } } }
I20260929 18:36:41.245082 17814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75bb6f7d5b8a448d98ebafeb6cc47b40 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:41.244853 17813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75bb6f7d5b8a448d98ebafeb6cc47b40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75bb6f7d5b8a448d98ebafeb6cc47b40. Latest consensus state: current_term: 1 leader_uuid: "75bb6f7d5b8a448d98ebafeb6cc47b40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75bb6f7d5b8a448d98ebafeb6cc47b40" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42709 } } }
I20260929 18:36:41.245520 17813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75bb6f7d5b8a448d98ebafeb6cc47b40 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:41.245641 17821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:36:41.249392 17821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:36:41.257138 17821 catalog_manager.cc:1347] Generated new cluster ID: e15775029827484b8b47e93a742d7cc6
I20260929 18:36:41.257234 17821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:36:41.266814 17821 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:36:41.268474 17821 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:36:41.280088 17821 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75bb6f7d5b8a448d98ebafeb6cc47b40: Generated new TSK 0
I20260929 18:36:41.281034 17821 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:36:41.291069 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 17734
2026-09-29T18:36:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1534 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:36:41 dist-test-slave-qcpv krb5kdc[17833](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:36:41 dist-test-slave-qcpv krb5kdc[17833](info): set up 2 sockets
Sep 29 18:36:41 dist-test-slave-qcpv krb5kdc[17833](info): commencing operation
krb5kdc: starting...
W20260929 18:36:42.371271 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:36:42 dist-test-slave-qcpv krb5kdc[17833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:36:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:36:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:36:42.515064 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36973
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:42.643654 17846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:42.644021 17846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:42.644101 17846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:42.648273 17846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:36:42.648398 17846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:42.648437 17846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:42.648474 17846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:36:42.648509 17846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:36:42.654102 17846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:42.655392 17846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:42.656502 17846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:42.665808 17852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:42.665901 17851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:42.665961 17854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:42.666407 17846 server_base.cc:1034] running on GCE node
I20260929 18:36:42.667079 17846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:42.668331 17846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:42.669544 17846 hybrid_clock.cc:648] HybridClock initialized: now 1790707002669510 us; error 59 us; skew 500 ppm
Sep 29 18:36:42 dist-test-slave-qcpv krb5kdc[17833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:42.673110 17846 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:36:42.674113 17846 webserver.cc:460] Webserver started at http://127.0.0.1:35021/ using document root <none> and password file <none>
I20260929 18:36:42.674705 17846 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:42.674772 17846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:42.675020 17846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:42.677083 17846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "3750c63aabf14b8895dbfd070858d8a4"
format_stamp: "Formatted at 2026-09-29 18:36:42 on dist-test-slave-qcpv"
server_key: "4b312e63456df91e196c711ab1f98776"
server_key_iv: "d5fd3ca8a7ee5f8b1959ba1a00608ccd"
server_key_version: "encryptionkey@0"
I20260929 18:36:42.677663 17846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "3750c63aabf14b8895dbfd070858d8a4"
format_stamp: "Formatted at 2026-09-29 18:36:42 on dist-test-slave-qcpv"
server_key: "4b312e63456df91e196c711ab1f98776"
server_key_iv: "d5fd3ca8a7ee5f8b1959ba1a00608ccd"
server_key_version: "encryptionkey@0"
I20260929 18:36:42.681720 17846 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 18:36:42.684907 17861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:42.686537 17846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 18:36:42.686681 17846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "3750c63aabf14b8895dbfd070858d8a4"
format_stamp: "Formatted at 2026-09-29 18:36:42 on dist-test-slave-qcpv"
server_key: "4b312e63456df91e196c711ab1f98776"
server_key_iv: "d5fd3ca8a7ee5f8b1959ba1a00608ccd"
server_key_version: "encryptionkey@0"
I20260929 18:36:42.686918 17846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:42.692246 17846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:42.696497 17846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:42.696738 17846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:42.705338 17846 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36973
I20260929 18:36:42.705426 17913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36973 every 8 connection(s)
I20260929 18:36:42.706396 17846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:36:42.710464 17914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:42.713897 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 17846
I20260929 18:36:42.714083 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:36:42.714390 29501 external_mini_cluster.cc:1428] Setting key 611b04496f47d33433465b309bd3ad5c
I20260929 18:36:42.718912 17914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3750c63aabf14b8895dbfd070858d8a4: Bootstrap starting.
I20260929 18:36:42.720803 17917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:42.716245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51255 (local address 127.0.0.1:36973)
0929 18:36:42.716755 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:42.716771 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:42.716808 (+ 37us) server_negotiation.cc:407] Connection header received
0929 18:36:42.717579 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:42.717599 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:42.717919 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:42.718259 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:42.719450 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:42.719456 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:42.719479 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:42.719490 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:42.719548 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:42.719570 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:42.719760 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:42.719836 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 18:36:42.720007 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":177,"threads_started":1}
I20260929 18:36:42.722175 17914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3750c63aabf14b8895dbfd070858d8a4: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:42.723268 17914 log.cc:824] T 00000000000000000000000000000000 P 3750c63aabf14b8895dbfd070858d8a4: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:42.724069 29501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:36:42.725854 17914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3750c63aabf14b8895dbfd070858d8a4: No bootstrap required, opened a new log
I20260929 18:36:42.729229 17917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:42.726911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32875 (local address 127.0.0.1:36973)
0929 18:36:42.727064 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:42.727069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:42.727219 (+ 150us) server_negotiation.cc:407] Connection header received
0929 18:36:42.727598 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:42.727603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:42.727726 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:42.727877 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:42.728767 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:42.728771 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:42.728773 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:42.728785 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:42.728814 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:42.728834 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:42.728932 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:42.729022 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 18:36:42.729068 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 18:36:42.729964 17914 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3750c63aabf14b8895dbfd070858d8a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3750c63aabf14b8895dbfd070858d8a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36973 } }
I20260929 18:36:42.730270 17914 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3750c63aabf14b8895dbfd070858d8a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:42.730326 17914 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3750c63aabf14b8895dbfd070858d8a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3750c63aabf14b8895dbfd070858d8a4, State: Initialized, Role: FOLLOWER
I20260929 18:36:42.730768 17914 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3750c63aabf14b8895dbfd070858d8a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3750c63aabf14b8895dbfd070858d8a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36973 } }
I20260929 18:36:42.730947 17914 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3750c63aabf14b8895dbfd070858d8a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:36:42.730995 17914 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3750c63aabf14b8895dbfd070858d8a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:36:42.731094 17914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3750c63aabf14b8895dbfd070858d8a4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:42.732131 17914 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3750c63aabf14b8895dbfd070858d8a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3750c63aabf14b8895dbfd070858d8a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36973 } }
I20260929 18:36:42.732482 17914 leader_election.cc:302] T 00000000000000000000000000000000 P 3750c63aabf14b8895dbfd070858d8a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3750c63aabf14b8895dbfd070858d8a4; no voters:
I20260929 18:36:42.732841 17914 leader_election.cc:288] T 00000000000000000000000000000000 P 3750c63aabf14b8895dbfd070858d8a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:36:42.733111 17924 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3750c63aabf14b8895dbfd070858d8a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:42.733448 17924 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3750c63aabf14b8895dbfd070858d8a4 [term 1 LEADER]: Becoming Leader. State: Replica: 3750c63aabf14b8895dbfd070858d8a4, State: Running, Role: LEADER
I20260929 18:36:42.733855 17924 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3750c63aabf14b8895dbfd070858d8a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3750c63aabf14b8895dbfd070858d8a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36973 } }
I20260929 18:36:42.734272 17914 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3750c63aabf14b8895dbfd070858d8a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:36:42.736269 17926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3750c63aabf14b8895dbfd070858d8a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3750c63aabf14b8895dbfd070858d8a4. Latest consensus state: current_term: 1 leader_uuid: "3750c63aabf14b8895dbfd070858d8a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3750c63aabf14b8895dbfd070858d8a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36973 } } }
I20260929 18:36:42.736456 17926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3750c63aabf14b8895dbfd070858d8a4 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:42.736878 17925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3750c63aabf14b8895dbfd070858d8a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3750c63aabf14b8895dbfd070858d8a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3750c63aabf14b8895dbfd070858d8a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36973 } } }
I20260929 18:36:42.737164 17925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3750c63aabf14b8895dbfd070858d8a4 [sys.catalog]: This master's current role is: LEADER
W20260929 18:36:42.741242 17939 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3750c63aabf14b8895dbfd070858d8a4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:36:42.741340 17939 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:36:42.741444 17938 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:36:42.742733 17938 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:36:42.750010 17938 catalog_manager.cc:1347] Generated new cluster ID: 114ad0f323604c56881d0472387e5bc1
I20260929 18:36:42.750116 17938 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:36:42.762431 17938 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:36:42.762933 17938 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:36:42.772923 17938 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3750c63aabf14b8895dbfd070858d8a4: Generated new TSK 0
I20260929 18:36:42.773829 17938 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:36:42.782229 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 17846
2026-09-29T18:36:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1492 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:36:42 dist-test-slave-qcpv krb5kdc[17945](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:36:42 dist-test-slave-qcpv krb5kdc[17945](info): set up 2 sockets
Sep 29 18:36:42 dist-test-slave-qcpv krb5kdc[17945](info): commencing operation
krb5kdc: starting...
W20260929 18:36:43.867388 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.009s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:36:43 dist-test-slave-qcpv krb5kdc[17945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:36:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:36:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:36:44.016170 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46485
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46485
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:44.169078 17958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:44.169490 17958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:44.169539 17958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:44.174845 17958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:36:44.175022 17958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:44.175047 17958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:44.175071 17958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:36:44.175091 17958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:36:44.181893 17958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:44.183715 17958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:44.185326 17958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:44.196821 17964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:44.196940 17963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:44.196821 17966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:44.197232 17958 server_base.cc:1034] running on GCE node
I20260929 18:36:44.198076 17958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:44.199451 17958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:44.200850 17958 hybrid_clock.cc:648] HybridClock initialized: now 1790707004200813 us; error 61 us; skew 500 ppm
Sep 29 18:36:44 dist-test-slave-qcpv krb5kdc[17945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707004, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:44.211843 17958 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:36:44.213054 17958 webserver.cc:460] Webserver started at http://127.0.0.1:34481/ using document root <none> and password file <none>
I20260929 18:36:44.213718 17958 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:44.213820 17958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:44.214066 17958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:44.216435 17958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "a20c5de9682c4b4ab45768920ad6be94"
format_stamp: "Formatted at 2026-09-29 18:36:44 on dist-test-slave-qcpv"
server_key: "5c506c73594b542c8b781be4ccd36a5b"
server_key_iv: "901cc6fe4a7505ced3170044c422f154"
server_key_version: "encryptionkey@0"
I20260929 18:36:44.217068 17958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "a20c5de9682c4b4ab45768920ad6be94"
format_stamp: "Formatted at 2026-09-29 18:36:44 on dist-test-slave-qcpv"
server_key: "5c506c73594b542c8b781be4ccd36a5b"
server_key_iv: "901cc6fe4a7505ced3170044c422f154"
server_key_version: "encryptionkey@0"
I20260929 18:36:44.221388 17958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260929 18:36:44.224480 17973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:44.226186 17958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 18:36:44.226388 17958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "a20c5de9682c4b4ab45768920ad6be94"
format_stamp: "Formatted at 2026-09-29 18:36:44 on dist-test-slave-qcpv"
server_key: "5c506c73594b542c8b781be4ccd36a5b"
server_key_iv: "901cc6fe4a7505ced3170044c422f154"
server_key_version: "encryptionkey@0"
I20260929 18:36:44.226527 17958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:44.245707 17958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:44.249429 17958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:44.249727 17958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:44.258654 17958 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46485
I20260929 18:36:44.258781 18025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46485 every 8 connection(s)
I20260929 18:36:44.259795 17958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:36:44.263243 18026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:44.266396 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 17958
I20260929 18:36:44.266595 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:36:44.266983 29501 external_mini_cluster.cc:1428] Setting key 767a465973617e06a15231cee6f94071
I20260929 18:36:44.271624 18026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a20c5de9682c4b4ab45768920ad6be94: Bootstrap starting.
I20260929 18:36:44.274628 18026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a20c5de9682c4b4ab45768920ad6be94: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:44.276463 18026 log.cc:824] T 00000000000000000000000000000000 P a20c5de9682c4b4ab45768920ad6be94: Log is configured to *not* fsync() on all Append() calls
Sep 29 18:36:44 dist-test-slave-qcpv krb5kdc[17945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 18:36:44.279285 18026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a20c5de9682c4b4ab45768920ad6be94: No bootstrap required, opened a new log
I20260929 18:36:44.283368 18026 raft_consensus.cc:348] T 00000000000000000000000000000000 P a20c5de9682c4b4ab45768920ad6be94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a20c5de9682c4b4ab45768920ad6be94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46485 } }
I20260929 18:36:44.283708 18026 raft_consensus.cc:374] T 00000000000000000000000000000000 P a20c5de9682c4b4ab45768920ad6be94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:44.283794 18026 raft_consensus.cc:729] T 00000000000000000000000000000000 P a20c5de9682c4b4ab45768920ad6be94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20c5de9682c4b4ab45768920ad6be94, State: Initialized, Role: FOLLOWER
I20260929 18:36:44.284394 18026 consensus_queue.cc:260] T 00000000000000000000000000000000 P a20c5de9682c4b4ab45768920ad6be94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a20c5de9682c4b4ab45768920ad6be94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46485 } }
I20260929 18:36:44.284591 18026 raft_consensus.cc:388] T 00000000000000000000000000000000 P a20c5de9682c4b4ab45768920ad6be94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:36:44.284657 18026 raft_consensus.cc:482] T 00000000000000000000000000000000 P a20c5de9682c4b4ab45768920ad6be94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:36:44.284778 18026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a20c5de9682c4b4ab45768920ad6be94 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:44.285650 18029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:44.268950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:35095 (local address 127.0.0.1:46485)
0929 18:36:44.269591 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:44.269609 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:44.269657 (+ 48us) server_negotiation.cc:407] Connection header received
0929 18:36:44.270566 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:44.270587 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:44.271145 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:44.271522 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:44.272387 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:44.273422 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:44.274328 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:44.274830 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:44.275066 (+ 236us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:44.278977 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:44.278984 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:44.279005 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:44.279046 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:44.281297 (+ 2251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:44.281989 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:44.281995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:44.282001 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:44.282156 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:44.282631 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:44.282634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:44.282636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:44.283393 (+ 757us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:44.283700 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:44.284196 (+ 496us) server_negotiation.cc:299] Negotiation successful
0929 18:36:44.284506 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":202,"threads_started":1}
I20260929 18:36:44.286283 18026 raft_consensus.cc:504] T 00000000000000000000000000000000 P a20c5de9682c4b4ab45768920ad6be94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a20c5de9682c4b4ab45768920ad6be94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46485 } }
I20260929 18:36:44.286674 18026 leader_election.cc:302] T 00000000000000000000000000000000 P a20c5de9682c4b4ab45768920ad6be94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a20c5de9682c4b4ab45768920ad6be94; no voters:
I20260929 18:36:44.287101 18026 leader_election.cc:288] T 00000000000000000000000000000000 P a20c5de9682c4b4ab45768920ad6be94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:36:44.287279 18031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a20c5de9682c4b4ab45768920ad6be94 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:44.287607 18031 raft_consensus.cc:686] T 00000000000000000000000000000000 P a20c5de9682c4b4ab45768920ad6be94 [term 1 LEADER]: Becoming Leader. State: Replica: a20c5de9682c4b4ab45768920ad6be94, State: Running, Role: LEADER
I20260929 18:36:44.288017 18031 consensus_queue.cc:237] T 00000000000000000000000000000000 P a20c5de9682c4b4ab45768920ad6be94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a20c5de9682c4b4ab45768920ad6be94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46485 } }
I20260929 18:36:44.288553 18026 sys_catalog.cc:564] T 00000000000000000000000000000000 P a20c5de9682c4b4ab45768920ad6be94 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:36:44.290381 29501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:36:44.290781 18032 sys_catalog.cc:455] T 00000000000000000000000000000000 P a20c5de9682c4b4ab45768920ad6be94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a20c5de9682c4b4ab45768920ad6be94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a20c5de9682c4b4ab45768920ad6be94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46485 } } }
I20260929 18:36:44.290966 18032 sys_catalog.cc:458] T 00000000000000000000000000000000 P a20c5de9682c4b4ab45768920ad6be94 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:44.291325 18033 sys_catalog.cc:455] T 00000000000000000000000000000000 P a20c5de9682c4b4ab45768920ad6be94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a20c5de9682c4b4ab45768920ad6be94. Latest consensus state: current_term: 1 leader_uuid: "a20c5de9682c4b4ab45768920ad6be94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a20c5de9682c4b4ab45768920ad6be94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46485 } } }
I20260929 18:36:44.291527 18033 sys_catalog.cc:458] T 00000000000000000000000000000000 P a20c5de9682c4b4ab45768920ad6be94 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:44.291949 18046 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:36:44.295437 18046 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:36:44.301757 18046 catalog_manager.cc:1347] Generated new cluster ID: 2cf42dda667c4dbb8dc74f153fd8d7da
I20260929 18:36:44.301841 18046 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:36:44.304863 18029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:44.293879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:47145 (local address 127.0.0.1:46485)
0929 18:36:44.294043 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:44.294050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:44.294515 (+ 465us) server_negotiation.cc:407] Connection header received
0929 18:36:44.294704 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:44.294707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:44.294770 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:44.294908 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:44.295986 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:44.296755 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:44.297967 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:44.298298 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:44.298404 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:44.300485 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:44.300490 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:44.300494 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:44.300528 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:44.302465 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:44.303430 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:44.303434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:44.303436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:44.303539 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:44.303885 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:44.303889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:44.303890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:44.304086 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:44.304271 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:44.304570 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 18:36:44.304685 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":43}
I20260929 18:36:44.317220 18046 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:36:44.318429 18046 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:36:44.324699 18046 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a20c5de9682c4b4ab45768920ad6be94: Generated new TSK 0
I20260929 18:36:44.325640 18046 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:36:44.336654 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 17958
W20260929 18:36:44.344089 17943 connection.cc:535] client connection to 127.0.0.1:46485 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46485): Connection reset by peer (error 104)
2026-09-29T18:36:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1553 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:36:44 dist-test-slave-qcpv krb5kdc[18057](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:36:44 dist-test-slave-qcpv krb5kdc[18057](info): set up 2 sockets
Sep 29 18:36:44 dist-test-slave-qcpv krb5kdc[18057](info): commencing operation
krb5kdc: starting...
W20260929 18:36:45.423477 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.009s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:36:45 dist-test-slave-qcpv krb5kdc[18057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:36:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:36:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:36:45.576539 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35335
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:45.699923 18070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:45.700258 18070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:45.700338 18070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:45.704564 18070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:36:45.704677 18070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:45.704715 18070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:45.704754 18070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:36:45.704787 18070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:36:45.710332 18070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:45.711572 18070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:45.712636 18070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:45.722141 18076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:45.722141 18075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:45.722318 18070 server_base.cc:1034] running on GCE node
W20260929 18:36:45.722505 18078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:45.723268 18070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:45.724625 18070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:45.727960 18070 hybrid_clock.cc:648] HybridClock initialized: now 1790707005727923 us; error 53 us; skew 500 ppm
Sep 29 18:36:45 dist-test-slave-qcpv krb5kdc[18057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:45.731140 18070 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:36:45.732115 18070 webserver.cc:460] Webserver started at http://127.0.0.1:42965/ using document root <none> and password file <none>
I20260929 18:36:45.732686 18070 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:45.732751 18070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:45.732925 18070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:45.735168 18070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "6e9bb96524b44c7bb13d96d18ee59b1e"
format_stamp: "Formatted at 2026-09-29 18:36:45 on dist-test-slave-qcpv"
server_key: "117734451af181bcdf18b1006d19bc2d"
server_key_iv: "43353941b95c6bb09094d719bb893436"
server_key_version: "encryptionkey@0"
I20260929 18:36:45.735716 18070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "6e9bb96524b44c7bb13d96d18ee59b1e"
format_stamp: "Formatted at 2026-09-29 18:36:45 on dist-test-slave-qcpv"
server_key: "117734451af181bcdf18b1006d19bc2d"
server_key_iv: "43353941b95c6bb09094d719bb893436"
server_key_version: "encryptionkey@0"
I20260929 18:36:45.740005 18070 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:36:45.742815 18085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:45.744699 18070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:36:45.744904 18070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "6e9bb96524b44c7bb13d96d18ee59b1e"
format_stamp: "Formatted at 2026-09-29 18:36:45 on dist-test-slave-qcpv"
server_key: "117734451af181bcdf18b1006d19bc2d"
server_key_iv: "43353941b95c6bb09094d719bb893436"
server_key_version: "encryptionkey@0"
I20260929 18:36:45.745072 18070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:45.759307 18070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:45.763242 18070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:45.763480 18070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:45.772305 18070 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35335
I20260929 18:36:45.772377 18137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35335 every 8 connection(s)
I20260929 18:36:45.773386 18070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:36:45.776404 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 18070
I20260929 18:36:45.776589 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:36:45.776934 29501 external_mini_cluster.cc:1428] Setting key 3b5d1e6f30dbab96f5329b2a47339607
I20260929 18:36:45.778283 18138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:45.786288 18138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e9bb96524b44c7bb13d96d18ee59b1e: Bootstrap starting.
I20260929 18:36:45.787223 18141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:45.778698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:48693 (local address 127.0.0.1:35335)
0929 18:36:45.779405 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:45.779438 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:45.779485 (+ 47us) server_negotiation.cc:407] Connection header received
0929 18:36:45.780213 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:45.780232 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:45.780628 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:45.780983 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:45.781403 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:45.782394 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:45.783381 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:45.783853 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:45.784027 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:45.785110 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:45.785115 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:45.785130 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:45.785651 (+ 521us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:45.785716 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:45.785741 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:45.785915 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:45.786119 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 18:36:45.786274 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":245,"threads_started":1}
I20260929 18:36:45.790046 18138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e9bb96524b44c7bb13d96d18ee59b1e: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:45.791036 29501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:36:45.791285 18138 log.cc:824] T 00000000000000000000000000000000 P 6e9bb96524b44c7bb13d96d18ee59b1e: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:45.794196 18138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e9bb96524b44c7bb13d96d18ee59b1e: No bootstrap required, opened a new log
I20260929 18:36:45.798822 18141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:45.793537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:41701 (local address 127.0.0.1:35335)
0929 18:36:45.793769 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:45.793774 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:45.794021 (+ 247us) server_negotiation.cc:407] Connection header received
0929 18:36:45.794149 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:45.794152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:45.794205 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:45.794319 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:45.794787 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:45.795574 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:45.796644 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:45.796999 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:45.797127 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:45.798227 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:45.798231 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:45.798234 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:45.798250 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:45.798284 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:45.798297 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:45.798415 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:45.798633 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 18:36:45.798676 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260929 18:36:45.798849 18138 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e9bb96524b44c7bb13d96d18ee59b1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e9bb96524b44c7bb13d96d18ee59b1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35335 } }
I20260929 18:36:45.799233 18138 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e9bb96524b44c7bb13d96d18ee59b1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:45.799320 18138 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e9bb96524b44c7bb13d96d18ee59b1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e9bb96524b44c7bb13d96d18ee59b1e, State: Initialized, Role: FOLLOWER
I20260929 18:36:45.799824 18138 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e9bb96524b44c7bb13d96d18ee59b1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e9bb96524b44c7bb13d96d18ee59b1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35335 } }
I20260929 18:36:45.800002 18138 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e9bb96524b44c7bb13d96d18ee59b1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:36:45.800098 18138 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e9bb96524b44c7bb13d96d18ee59b1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:36:45.800204 18138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e9bb96524b44c7bb13d96d18ee59b1e [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:45.801468 18138 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e9bb96524b44c7bb13d96d18ee59b1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e9bb96524b44c7bb13d96d18ee59b1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35335 } }
I20260929 18:36:45.801882 18138 leader_election.cc:302] T 00000000000000000000000000000000 P 6e9bb96524b44c7bb13d96d18ee59b1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e9bb96524b44c7bb13d96d18ee59b1e; no voters:
I20260929 18:36:45.802307 18138 leader_election.cc:288] T 00000000000000000000000000000000 P 6e9bb96524b44c7bb13d96d18ee59b1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:36:45.802615 18148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e9bb96524b44c7bb13d96d18ee59b1e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:45.802975 18148 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e9bb96524b44c7bb13d96d18ee59b1e [term 1 LEADER]: Becoming Leader. State: Replica: 6e9bb96524b44c7bb13d96d18ee59b1e, State: Running, Role: LEADER
I20260929 18:36:45.803375 18148 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e9bb96524b44c7bb13d96d18ee59b1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e9bb96524b44c7bb13d96d18ee59b1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35335 } }
I20260929 18:36:45.803802 18138 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e9bb96524b44c7bb13d96d18ee59b1e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:36:45.806162 18150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e9bb96524b44c7bb13d96d18ee59b1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e9bb96524b44c7bb13d96d18ee59b1e. Latest consensus state: current_term: 1 leader_uuid: "6e9bb96524b44c7bb13d96d18ee59b1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e9bb96524b44c7bb13d96d18ee59b1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35335 } } }
I20260929 18:36:45.806212 18149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e9bb96524b44c7bb13d96d18ee59b1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e9bb96524b44c7bb13d96d18ee59b1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e9bb96524b44c7bb13d96d18ee59b1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35335 } } }
I20260929 18:36:45.806370 18150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e9bb96524b44c7bb13d96d18ee59b1e [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:45.806375 18149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e9bb96524b44c7bb13d96d18ee59b1e [sys.catalog]: This master's current role is: LEADER
W20260929 18:36:45.810845 18162 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6e9bb96524b44c7bb13d96d18ee59b1e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:36:45.811050 18162 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:36:45.811131 18163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:36:45.812040 18163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:36:45.818720 18163 catalog_manager.cc:1347] Generated new cluster ID: 5506cd61a4e84306b5919fc8c3b3bef3
I20260929 18:36:45.818828 18163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:36:45.833968 18163 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:36:45.835721 18163 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:36:45.845078 18163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e9bb96524b44c7bb13d96d18ee59b1e: Generated new TSK 0
I20260929 18:36:45.846150 18163 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:36:45.860838 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 18070
W20260929 18:36:45.867027 18055 connection.cc:535] client connection to 127.0.0.1:35335 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35335): Connection reset by peer (error 104)
2026-09-29T18:36:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1522 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:36:45 dist-test-slave-qcpv krb5kdc[18169](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:36:45 dist-test-slave-qcpv krb5kdc[18169](info): set up 2 sockets
Sep 29 18:36:45 dist-test-slave-qcpv krb5kdc[18169](info): commencing operation
krb5kdc: starting...
W20260929 18:36:46.923346 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:36:47 dist-test-slave-qcpv krb5kdc[18169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:36:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:36:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:36:47.071257 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43591
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:47.214995 18182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:47.215446 18182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:47.215536 18182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:47.222719 18182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:36:47.222947 18182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:47.223006 18182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:47.223048 18182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:36:47.223107 18182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:36:47.231492 18182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:47.233448 18182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:47.235240 18182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:36:47.248142 18182 server_base.cc:1034] running on GCE node
W20260929 18:36:47.248179 18190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:47.248198 18188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:47.248369 18187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:47.249316 18182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:47.250736 18182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:47.251964 18182 hybrid_clock.cc:648] HybridClock initialized: now 1790707007251942 us; error 56 us; skew 500 ppm
Sep 29 18:36:47 dist-test-slave-qcpv krb5kdc[18169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:47.255699 18182 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:36:47.256882 18182 webserver.cc:460] Webserver started at http://127.0.0.1:38899/ using document root <none> and password file <none>
I20260929 18:36:47.257743 18182 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:47.257871 18182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:47.258150 18182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:47.260746 18182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "62e03dc39a4748cdb403222638e7a686"
format_stamp: "Formatted at 2026-09-29 18:36:47 on dist-test-slave-qcpv"
server_key: "bd76a06658af4977194cbcfde981aaef"
server_key_iv: "84e741e2e39264d170758d62045a2697"
server_key_version: "encryptionkey@0"
I20260929 18:36:47.261411 18182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "62e03dc39a4748cdb403222638e7a686"
format_stamp: "Formatted at 2026-09-29 18:36:47 on dist-test-slave-qcpv"
server_key: "bd76a06658af4977194cbcfde981aaef"
server_key_iv: "84e741e2e39264d170758d62045a2697"
server_key_version: "encryptionkey@0"
I20260929 18:36:47.265810 18182 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 18:36:47.268967 18197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:47.270715 18182 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:36:47.270947 18182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "62e03dc39a4748cdb403222638e7a686"
format_stamp: "Formatted at 2026-09-29 18:36:47 on dist-test-slave-qcpv"
server_key: "bd76a06658af4977194cbcfde981aaef"
server_key_iv: "84e741e2e39264d170758d62045a2697"
server_key_version: "encryptionkey@0"
I20260929 18:36:47.271139 18182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:47.277328 18182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:47.281342 18182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:47.281595 18182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:47.290037 18182 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43591
I20260929 18:36:47.290135 18249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43591 every 8 connection(s)
I20260929 18:36:47.291213 18182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:36:47.295440 18250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:47.300724 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 18182
I20260929 18:36:47.300892 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:36:47.301237 29501 external_mini_cluster.cc:1428] Setting key 975c8a4c7285635d336696d7c3ab80c5
I20260929 18:36:47.303184 18250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62e03dc39a4748cdb403222638e7a686: Bootstrap starting.
I20260929 18:36:47.306260 18250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62e03dc39a4748cdb403222638e7a686: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:47.307711 18250 log.cc:824] T 00000000000000000000000000000000 P 62e03dc39a4748cdb403222638e7a686: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:47.308102 18253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:47.303227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:44523 (local address 127.0.0.1:43591)
0929 18:36:47.303798 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:47.303814 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:47.303861 (+ 47us) server_negotiation.cc:407] Connection header received
0929 18:36:47.304687 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:47.304705 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:47.305047 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:47.305390 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:47.306502 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:47.306508 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:47.306519 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:47.306534 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:47.306607 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:47.306629 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:47.306743 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:47.307025 (+ 282us) server_negotiation.cc:299] Negotiation successful
0929 18:36:47.307236 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":184,"threads_started":1}
I20260929 18:36:47.310591 18250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62e03dc39a4748cdb403222638e7a686: No bootstrap required, opened a new log
I20260929 18:36:47.311379 29501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:36:47.316561 18253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:47.314303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:36439 (local address 127.0.0.1:43591)
0929 18:36:47.314484 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:47.314492 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:47.314801 (+ 309us) server_negotiation.cc:407] Connection header received
0929 18:36:47.314984 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:47.314987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:47.315025 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:47.315107 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:47.316030 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:47.316034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:47.316036 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:47.316047 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:47.316075 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:47.316086 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:47.316171 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:47.316288 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 18:36:47.316327 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 18:36:47.316320 18250 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62e03dc39a4748cdb403222638e7a686 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62e03dc39a4748cdb403222638e7a686" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43591 } }
I20260929 18:36:47.316728 18250 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62e03dc39a4748cdb403222638e7a686 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:47.316810 18250 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62e03dc39a4748cdb403222638e7a686 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62e03dc39a4748cdb403222638e7a686, State: Initialized, Role: FOLLOWER
I20260929 18:36:47.317355 18250 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62e03dc39a4748cdb403222638e7a686 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62e03dc39a4748cdb403222638e7a686" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43591 } }
I20260929 18:36:47.317505 18250 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62e03dc39a4748cdb403222638e7a686 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:36:47.317601 18250 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62e03dc39a4748cdb403222638e7a686 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:36:47.317746 18250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62e03dc39a4748cdb403222638e7a686 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:47.319024 18250 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62e03dc39a4748cdb403222638e7a686 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62e03dc39a4748cdb403222638e7a686" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43591 } }
I20260929 18:36:47.319407 18250 leader_election.cc:302] T 00000000000000000000000000000000 P 62e03dc39a4748cdb403222638e7a686 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62e03dc39a4748cdb403222638e7a686; no voters:
I20260929 18:36:47.319748 18250 leader_election.cc:288] T 00000000000000000000000000000000 P 62e03dc39a4748cdb403222638e7a686 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:36:47.319993 18260 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62e03dc39a4748cdb403222638e7a686 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:47.320508 18260 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62e03dc39a4748cdb403222638e7a686 [term 1 LEADER]: Becoming Leader. State: Replica: 62e03dc39a4748cdb403222638e7a686, State: Running, Role: LEADER
I20260929 18:36:47.320935 18260 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62e03dc39a4748cdb403222638e7a686 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62e03dc39a4748cdb403222638e7a686" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43591 } }
I20260929 18:36:47.321350 18250 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62e03dc39a4748cdb403222638e7a686 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:36:47.323068 18261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62e03dc39a4748cdb403222638e7a686 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62e03dc39a4748cdb403222638e7a686" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62e03dc39a4748cdb403222638e7a686" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43591 } } }
I20260929 18:36:47.323092 18262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62e03dc39a4748cdb403222638e7a686 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62e03dc39a4748cdb403222638e7a686. Latest consensus state: current_term: 1 leader_uuid: "62e03dc39a4748cdb403222638e7a686" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62e03dc39a4748cdb403222638e7a686" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43591 } } }
I20260929 18:36:47.323264 18261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62e03dc39a4748cdb403222638e7a686 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:47.323264 18262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62e03dc39a4748cdb403222638e7a686 [sys.catalog]: This master's current role is: LEADER
W20260929 18:36:47.327493 18275 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 62e03dc39a4748cdb403222638e7a686: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:36:47.327621 18275 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:36:47.327724 18272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:36:47.328583 18272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:36:47.336123 18272 catalog_manager.cc:1347] Generated new cluster ID: 9f6d8c52d335443b849fbf1124ed50cb
I20260929 18:36:47.336524 18272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:36:47.354437 18272 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:36:47.355000 18272 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:36:47.369930 18272 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62e03dc39a4748cdb403222638e7a686: Generated new TSK 0
I20260929 18:36:47.371030 18272 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:36:47.381990 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 18182
2026-09-29T18:36:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1520 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:36:47 dist-test-slave-qcpv krb5kdc[18281](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:36:47 dist-test-slave-qcpv krb5kdc[18281](info): set up 2 sockets
Sep 29 18:36:47 dist-test-slave-qcpv krb5kdc[18281](info): commencing operation
krb5kdc: starting...
W20260929 18:36:48.479391 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:36:48 dist-test-slave-qcpv krb5kdc[18281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:36:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:36:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:36:48.623994 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36517
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:48.760113 18294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:48.760417 18294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:48.760459 18294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:48.764667 18294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:36:48.764745 18294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:48.764766 18294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:48.764783 18294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:36:48.764799 18294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:36:48.770473 18294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36517
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:48.771734 18294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:48.772847 18294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:48.783937 18299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:48.784039 18300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:48.783960 18302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:48.784263 18294 server_base.cc:1034] running on GCE node
I20260929 18:36:48.785106 18294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:48.786432 18294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:48.787672 18294 hybrid_clock.cc:648] HybridClock initialized: now 1790707008787621 us; error 67 us; skew 500 ppm
Sep 29 18:36:48 dist-test-slave-qcpv krb5kdc[18281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:48.790979 18294 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:36:48.792228 18294 webserver.cc:460] Webserver started at http://127.0.0.1:36565/ using document root <none> and password file <none>
I20260929 18:36:48.792857 18294 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:48.792958 18294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:48.793227 18294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:48.795639 18294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "39400f36e88e45d58d122df1bc1be161"
format_stamp: "Formatted at 2026-09-29 18:36:48 on dist-test-slave-qcpv"
server_key: "9d355649986499c93fe2b723f41f7207"
server_key_iv: "d976bb58fc499f1ed5b92bd29ea7f7d1"
server_key_version: "encryptionkey@0"
I20260929 18:36:48.796288 18294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "39400f36e88e45d58d122df1bc1be161"
format_stamp: "Formatted at 2026-09-29 18:36:48 on dist-test-slave-qcpv"
server_key: "9d355649986499c93fe2b723f41f7207"
server_key_iv: "d976bb58fc499f1ed5b92bd29ea7f7d1"
server_key_version: "encryptionkey@0"
I20260929 18:36:48.800926 18294 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:36:48.804237 18309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:48.806087 18294 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:36:48.806285 18294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "39400f36e88e45d58d122df1bc1be161"
format_stamp: "Formatted at 2026-09-29 18:36:48 on dist-test-slave-qcpv"
server_key: "9d355649986499c93fe2b723f41f7207"
server_key_iv: "d976bb58fc499f1ed5b92bd29ea7f7d1"
server_key_version: "encryptionkey@0"
I20260929 18:36:48.806427 18294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:48.819921 18294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:48.824010 18294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:48.824270 18294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:48.832921 18294 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36517
I20260929 18:36:48.832955 18361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36517 every 8 connection(s)
I20260929 18:36:48.834204 18294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:36:48.838544 18362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:48.843214 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 18294
I20260929 18:36:48.843348 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:36:48.843688 29501 external_mini_cluster.cc:1428] Setting key b71f7c63b24eb3e315c89d09de35582d
I20260929 18:36:48.846485 18362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39400f36e88e45d58d122df1bc1be161: Bootstrap starting.
I20260929 18:36:48.851148 18362 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39400f36e88e45d58d122df1bc1be161: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:48.852208 18362 log.cc:824] T 00000000000000000000000000000000 P 39400f36e88e45d58d122df1bc1be161: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:48.855381 18362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39400f36e88e45d58d122df1bc1be161: No bootstrap required, opened a new log
Sep 29 18:36:48 dist-test-slave-qcpv krb5kdc[18281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 18:36:48.860664 18362 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39400f36e88e45d58d122df1bc1be161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39400f36e88e45d58d122df1bc1be161" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36517 } }
I20260929 18:36:48.860999 18362 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39400f36e88e45d58d122df1bc1be161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:48.861079 18362 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39400f36e88e45d58d122df1bc1be161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39400f36e88e45d58d122df1bc1be161, State: Initialized, Role: FOLLOWER
I20260929 18:36:48.861681 18362 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39400f36e88e45d58d122df1bc1be161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39400f36e88e45d58d122df1bc1be161" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36517 } }
I20260929 18:36:48.861881 18362 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39400f36e88e45d58d122df1bc1be161 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:36:48.861981 18362 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39400f36e88e45d58d122df1bc1be161 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:36:48.862111 18362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39400f36e88e45d58d122df1bc1be161 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:48.863694 18362 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39400f36e88e45d58d122df1bc1be161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39400f36e88e45d58d122df1bc1be161" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36517 } }
I20260929 18:36:48.864215 18362 leader_election.cc:302] T 00000000000000000000000000000000 P 39400f36e88e45d58d122df1bc1be161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39400f36e88e45d58d122df1bc1be161; no voters:
I20260929 18:36:48.864961 18362 leader_election.cc:288] T 00000000000000000000000000000000 P 39400f36e88e45d58d122df1bc1be161 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:36:48.864970 18365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:48.845621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:36725 (local address 127.0.0.1:36517)
0929 18:36:48.846198 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:48.846226 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:48.846291 (+ 65us) server_negotiation.cc:407] Connection header received
0929 18:36:48.847415 (+ 1124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:48.847444 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:48.847984 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:48.848961 (+ 977us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:48.850062 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:48.851712 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:48.852945 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:48.853835 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:48.854025 (+ 190us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:48.857314 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:48.857322 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:48.857347 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:48.857398 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:48.860501 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:48.861456 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:48.861467 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:48.861476 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:48.861697 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:48.862154 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:48.862157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:48.862158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:48.862820 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:48.863168 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:48.863481 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 18:36:48.863755 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":380,"thread_start_us":160,"threads_started":1}
I20260929 18:36:48.865059 18367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39400f36e88e45d58d122df1bc1be161 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:48.865283 18367 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39400f36e88e45d58d122df1bc1be161 [term 1 LEADER]: Becoming Leader. State: Replica: 39400f36e88e45d58d122df1bc1be161, State: Running, Role: LEADER
I20260929 18:36:48.865805 18367 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39400f36e88e45d58d122df1bc1be161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39400f36e88e45d58d122df1bc1be161" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36517 } }
I20260929 18:36:48.866915 18362 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39400f36e88e45d58d122df1bc1be161 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:36:48.867662 18369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39400f36e88e45d58d122df1bc1be161 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39400f36e88e45d58d122df1bc1be161. Latest consensus state: current_term: 1 leader_uuid: "39400f36e88e45d58d122df1bc1be161" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39400f36e88e45d58d122df1bc1be161" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36517 } } }
I20260929 18:36:48.867856 18369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39400f36e88e45d58d122df1bc1be161 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:48.868613 18368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39400f36e88e45d58d122df1bc1be161 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39400f36e88e45d58d122df1bc1be161" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39400f36e88e45d58d122df1bc1be161" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36517 } } }
I20260929 18:36:48.868732 18368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39400f36e88e45d58d122df1bc1be161 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:48.869023 29501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:36:48.869210 18373 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:36:48.873482 18373 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:36:48.880923 18373 catalog_manager.cc:1347] Generated new cluster ID: 94b549b31b314dc8804e8e4ea9a39198
I20260929 18:36:48.881040 18373 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:36:48.892910 18365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:48.871870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:40741 (local address 127.0.0.1:36517)
0929 18:36:48.871990 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:48.871996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:48.876423 (+ 4427us) server_negotiation.cc:407] Connection header received
0929 18:36:48.876612 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:48.876616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:48.876684 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:48.876884 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:48.877961 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:48.879098 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:48.880735 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:48.881101 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:48.881203 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:48.888816 (+ 7613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:48.888820 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:48.888822 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:48.888854 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:48.890559 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:48.891426 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:48.891432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:48.891439 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:48.891531 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:48.891907 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:48.891910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:48.891911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:48.892095 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:48.892283 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:48.892597 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 18:36:48.892710 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 18:36:48.934368 18373 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:36:48.936110 18373 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:36:48.946237 18373 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39400f36e88e45d58d122df1bc1be161: Generated new TSK 0
I20260929 18:36:48.947551 18373 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:36:48.964385 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 18294
W20260929 18:36:48.971119 18279 connection.cc:535] client connection to 127.0.0.1:36517 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36517): Connection reset by peer (error 104)
2026-09-29T18:36:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1583 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:36:49 dist-test-slave-qcpv krb5kdc[18393](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:36:49 dist-test-slave-qcpv krb5kdc[18393](info): set up 2 sockets
Sep 29 18:36:49 dist-test-slave-qcpv krb5kdc[18393](info): commencing operation
krb5kdc: starting...
W20260929 18:36:50.064047 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:36:50 dist-test-slave-qcpv krb5kdc[18393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:36:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:36:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:36:50.219518 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40065
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:50.336895 18406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:50.337193 18406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:50.337239 18406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:50.341358 18406 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:36:50.341570 18406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:50.341620 18406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:50.341639 18406 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:36:50.341658 18406 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:36:50.348266 18406 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40065
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:50.349442 18406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:50.350472 18406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:50.359871 18412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:50.359896 18411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:50.359871 18414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:50.360241 18406 server_base.cc:1034] running on GCE node
I20260929 18:36:50.361186 18406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:50.362819 18406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:50.364108 18406 hybrid_clock.cc:648] HybridClock initialized: now 1790707010364070 us; error 61 us; skew 500 ppm
Sep 29 18:36:50 dist-test-slave-qcpv krb5kdc[18393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707010, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:50.375775 18406 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:36:50.376899 18406 webserver.cc:460] Webserver started at http://127.0.0.1:43695/ using document root <none> and password file <none>
I20260929 18:36:50.377660 18406 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:50.377758 18406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:50.377949 18406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:50.380621 18406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "b6e941584b5b4b9a87bc021548370957"
format_stamp: "Formatted at 2026-09-29 18:36:50 on dist-test-slave-qcpv"
server_key: "e616ed749288460aebe6d38aa50da262"
server_key_iv: "170bf8d25584cb781c6f09c2a2b96618"
server_key_version: "encryptionkey@0"
I20260929 18:36:50.381212 18406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "b6e941584b5b4b9a87bc021548370957"
format_stamp: "Formatted at 2026-09-29 18:36:50 on dist-test-slave-qcpv"
server_key: "e616ed749288460aebe6d38aa50da262"
server_key_iv: "170bf8d25584cb781c6f09c2a2b96618"
server_key_version: "encryptionkey@0"
I20260929 18:36:50.385867 18406 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:36:50.389091 18421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:50.390848 18406 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:36:50.391191 18406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "b6e941584b5b4b9a87bc021548370957"
format_stamp: "Formatted at 2026-09-29 18:36:50 on dist-test-slave-qcpv"
server_key: "e616ed749288460aebe6d38aa50da262"
server_key_iv: "170bf8d25584cb781c6f09c2a2b96618"
server_key_version: "encryptionkey@0"
I20260929 18:36:50.391332 18406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:50.403316 18406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:50.406670 18406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:50.406867 18406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:50.415954 18406 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40065
I20260929 18:36:50.416039 18473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40065 every 8 connection(s)
I20260929 18:36:50.417153 18406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:36:50.421001 18474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:50.427525 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 18406
I20260929 18:36:50.427644 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:36:50.427960 29501 external_mini_cluster.cc:1428] Setting key cc3cc75eb8a26c20c1ccf9a08f278848
I20260929 18:36:50.429315 18474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6e941584b5b4b9a87bc021548370957: Bootstrap starting.
I20260929 18:36:50.432407 18474 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6e941584b5b4b9a87bc021548370957: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:50.433311 18474 log.cc:824] T 00000000000000000000000000000000 P b6e941584b5b4b9a87bc021548370957: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:50.435941 18474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6e941584b5b4b9a87bc021548370957: No bootstrap required, opened a new log
I20260929 18:36:50.439347 18477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:50.429865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:53883 (local address 127.0.0.1:40065)
0929 18:36:50.430357 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:50.430375 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:50.430411 (+ 36us) server_negotiation.cc:407] Connection header received
0929 18:36:50.431288 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:50.431308 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:50.431687 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:50.432010 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:50.432660 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:50.433778 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:50.435000 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:50.435661 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:50.435842 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:50.437051 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:50.437056 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:50.437071 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:50.437591 (+ 520us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:50.437656 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:50.437686 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:50.437835 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:50.438129 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 18:36:50.438266 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":177,"threads_started":1}
I20260929 18:36:50.440717 18474 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6e941584b5b4b9a87bc021548370957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6e941584b5b4b9a87bc021548370957" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40065 } }
I20260929 18:36:50.440943 18474 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6e941584b5b4b9a87bc021548370957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:50.440975 18474 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6e941584b5b4b9a87bc021548370957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6e941584b5b4b9a87bc021548370957, State: Initialized, Role: FOLLOWER
I20260929 18:36:50.441416 18474 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6e941584b5b4b9a87bc021548370957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6e941584b5b4b9a87bc021548370957" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40065 } }
I20260929 18:36:50.441553 18474 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6e941584b5b4b9a87bc021548370957 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:36:50.441594 18474 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6e941584b5b4b9a87bc021548370957 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:36:50.441721 18474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6e941584b5b4b9a87bc021548370957 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:50.442992 18474 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6e941584b5b4b9a87bc021548370957 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6e941584b5b4b9a87bc021548370957" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40065 } }
I20260929 18:36:50.443347 18474 leader_election.cc:302] T 00000000000000000000000000000000 P b6e941584b5b4b9a87bc021548370957 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6e941584b5b4b9a87bc021548370957; no voters:
I20260929 18:36:50.443595 29501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:36:50.443779 18474 leader_election.cc:288] T 00000000000000000000000000000000 P b6e941584b5b4b9a87bc021548370957 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:36:50.443925 18479 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6e941584b5b4b9a87bc021548370957 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:50.444187 18479 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6e941584b5b4b9a87bc021548370957 [term 1 LEADER]: Becoming Leader. State: Replica: b6e941584b5b4b9a87bc021548370957, State: Running, Role: LEADER
I20260929 18:36:50.444664 18479 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6e941584b5b4b9a87bc021548370957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6e941584b5b4b9a87bc021548370957" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40065 } }
I20260929 18:36:50.445148 18474 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6e941584b5b4b9a87bc021548370957 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:36:50.446664 18479 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6e941584b5b4b9a87bc021548370957 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6e941584b5b4b9a87bc021548370957. Latest consensus state: current_term: 1 leader_uuid: "b6e941584b5b4b9a87bc021548370957" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6e941584b5b4b9a87bc021548370957" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40065 } } }
I20260929 18:36:50.446831 18479 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6e941584b5b4b9a87bc021548370957 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:50.447175 18481 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6e941584b5b4b9a87bc021548370957 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6e941584b5b4b9a87bc021548370957" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6e941584b5b4b9a87bc021548370957" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40065 } } }
I20260929 18:36:50.447299 18481 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6e941584b5b4b9a87bc021548370957 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:50.447345 18495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:36:50.451654 18495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:36:50.452412 18477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:50.447200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:41897 (local address 127.0.0.1:40065)
0929 18:36:50.447329 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:50.447335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:50.447631 (+ 296us) server_negotiation.cc:407] Connection header received
0929 18:36:50.447838 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:50.447841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:50.447897 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:50.447951 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:50.448454 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:50.449214 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:50.450349 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:50.450692 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:50.450824 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:50.451744 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:50.451754 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:50.451757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:50.451780 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:50.451804 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:50.451814 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:50.451913 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:50.452245 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 18:36:50.452294 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:36:50.458734 18495 catalog_manager.cc:1347] Generated new cluster ID: f4c0b33168f548e0bf68089cb6f4237d
I20260929 18:36:50.458889 18495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:36:50.484908 18495 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:36:50.486218 18495 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:36:50.493837 18495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6e941584b5b4b9a87bc021548370957: Generated new TSK 0
I20260929 18:36:50.494709 18495 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:36:50.501726 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 18406
W20260929 18:36:50.509126 18391 connection.cc:535] client connection to 127.0.0.1:40065 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40065): Connection reset by peer (error 104)
2026-09-29T18:36:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1537 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:36:50 dist-test-slave-qcpv krb5kdc[18505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:36:50 dist-test-slave-qcpv krb5kdc[18505](info): set up 2 sockets
Sep 29 18:36:50 dist-test-slave-qcpv krb5kdc[18505](info): commencing operation
krb5kdc: starting...
W20260929 18:36:51.571871 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:36:51 dist-test-slave-qcpv krb5kdc[18505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:36:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:36:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 18:36:51.719028 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46269
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:51.845973 18518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:51.846344 18518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:51.846428 18518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:51.850677 18518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:36:51.850801 18518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:51.850839 18518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:51.850899 18518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:36:51.850956 18518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:36:51.856926 18518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46269
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706718260899-29501-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:51.858147 18518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:51.859338 18518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:51.868539 18524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:51.868546 18523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:51.868678 18526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:51.869038 18518 server_base.cc:1034] running on GCE node
I20260929 18:36:51.869604 18518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:51.870769 18518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:51.872068 18518 hybrid_clock.cc:648] HybridClock initialized: now 1790707011872019 us; error 70 us; skew 500 ppm
Sep 29 18:36:51 dist-test-slave-qcpv krb5kdc[18505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:51.875859 18518 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 18:36:51.877007 18518 webserver.cc:460] Webserver started at http://127.0.0.1:39397/ using document root <none> and password file <none>
I20260929 18:36:51.877568 18518 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:51.877632 18518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:51.877802 18518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:51.879935 18518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "f4fae354286e47b8acf71dfc8726b00a"
format_stamp: "Formatted at 2026-09-29 18:36:51 on dist-test-slave-qcpv"
server_key: "afa7940cb07dd07f61342e35a1b25b42"
server_key_iv: "d51f32cb73c0e46db290d1c12af99795"
server_key_version: "encryptionkey@0"
I20260929 18:36:51.880461 18518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "f4fae354286e47b8acf71dfc8726b00a"
format_stamp: "Formatted at 2026-09-29 18:36:51 on dist-test-slave-qcpv"
server_key: "afa7940cb07dd07f61342e35a1b25b42"
server_key_iv: "d51f32cb73c0e46db290d1c12af99795"
server_key_version: "encryptionkey@0"
I20260929 18:36:51.884603 18518 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:36:51.887633 18533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:51.889217 18518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:36:51.889355 18518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "f4fae354286e47b8acf71dfc8726b00a"
format_stamp: "Formatted at 2026-09-29 18:36:51 on dist-test-slave-qcpv"
server_key: "afa7940cb07dd07f61342e35a1b25b42"
server_key_iv: "d51f32cb73c0e46db290d1c12af99795"
server_key_version: "encryptionkey@0"
I20260929 18:36:51.889459 18518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:51.894232 18518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:51.897578 18518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:51.897750 18518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:51.905419 18518 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46269
I20260929 18:36:51.905601 18585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46269 every 8 connection(s)
I20260929 18:36:51.906433 18518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:36:51.907766 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 18518
I20260929 18:36:51.907939 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:36:51.908315 29501 external_mini_cluster.cc:1428] Setting key 858dbe269a57fa554b1e041f8b987168
I20260929 18:36:51.910588 18586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:51.914383 18589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:51.910103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:50331 (local address 127.0.0.1:46269)
0929 18:36:51.910646 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:51.910688 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:51.910739 (+ 51us) server_negotiation.cc:407] Connection header received
0929 18:36:51.911541 (+ 802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:51.911561 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:51.911883 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:51.912257 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:51.913120 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:51.913125 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:51.913136 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:51.913147 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:51.913207 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:51.913227 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:51.913389 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:51.913465 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 18:36:51.913633 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":179,"threads_started":1}
I20260929 18:36:51.917831 29501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:36:51.919082 18586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4fae354286e47b8acf71dfc8726b00a: Bootstrap starting.
I20260929 18:36:51.922993 18589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:51.920409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.12:40383 (local address 127.0.0.1:46269)
0929 18:36:51.920541 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:51.920544 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:51.921326 (+ 782us) server_negotiation.cc:407] Connection header received
0929 18:36:51.921504 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:51.921507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:51.921545 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:51.921634 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:51.922489 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:51.922493 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:51.922495 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:51.922505 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:51.922532 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:51.922542 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:51.922626 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:51.922741 (+ 115us) server_negotiation.cc:299] Negotiation successful
0929 18:36:51.922780 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:36:51.923295 18586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4fae354286e47b8acf71dfc8726b00a: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:51.924381 18586 log.cc:824] T 00000000000000000000000000000000 P f4fae354286e47b8acf71dfc8726b00a: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:51.927263 18586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4fae354286e47b8acf71dfc8726b00a: No bootstrap required, opened a new log
I20260929 18:36:51.932266 18586 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4fae354286e47b8acf71dfc8726b00a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4fae354286e47b8acf71dfc8726b00a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46269 } }
I20260929 18:36:51.932590 18586 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4fae354286e47b8acf71dfc8726b00a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:51.932664 18586 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4fae354286e47b8acf71dfc8726b00a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4fae354286e47b8acf71dfc8726b00a, State: Initialized, Role: FOLLOWER
I20260929 18:36:51.933207 18586 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4fae354286e47b8acf71dfc8726b00a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4fae354286e47b8acf71dfc8726b00a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46269 } }
I20260929 18:36:51.933342 18586 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4fae354286e47b8acf71dfc8726b00a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:36:51.933449 18586 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4fae354286e47b8acf71dfc8726b00a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:36:51.933599 18586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4fae354286e47b8acf71dfc8726b00a [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:51.935071 18586 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4fae354286e47b8acf71dfc8726b00a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4fae354286e47b8acf71dfc8726b00a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46269 } }
I20260929 18:36:51.935593 18586 leader_election.cc:302] T 00000000000000000000000000000000 P f4fae354286e47b8acf71dfc8726b00a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4fae354286e47b8acf71dfc8726b00a; no voters:
I20260929 18:36:51.936060 18586 leader_election.cc:288] T 00000000000000000000000000000000 P f4fae354286e47b8acf71dfc8726b00a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:36:51.936400 18596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4fae354286e47b8acf71dfc8726b00a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:51.936805 18596 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4fae354286e47b8acf71dfc8726b00a [term 1 LEADER]: Becoming Leader. State: Replica: f4fae354286e47b8acf71dfc8726b00a, State: Running, Role: LEADER
I20260929 18:36:51.937285 18596 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4fae354286e47b8acf71dfc8726b00a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4fae354286e47b8acf71dfc8726b00a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46269 } }
I20260929 18:36:51.937716 18586 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4fae354286e47b8acf71dfc8726b00a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:36:51.939224 18597 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4fae354286e47b8acf71dfc8726b00a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4fae354286e47b8acf71dfc8726b00a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4fae354286e47b8acf71dfc8726b00a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46269 } } }
I20260929 18:36:51.939276 18598 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4fae354286e47b8acf71dfc8726b00a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4fae354286e47b8acf71dfc8726b00a. Latest consensus state: current_term: 1 leader_uuid: "f4fae354286e47b8acf71dfc8726b00a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4fae354286e47b8acf71dfc8726b00a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46269 } } }
I20260929 18:36:51.939385 18597 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4fae354286e47b8acf71dfc8726b00a [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:51.939385 18598 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4fae354286e47b8acf71dfc8726b00a [sys.catalog]: This master's current role is: LEADER
W20260929 18:36:51.943269 18611 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f4fae354286e47b8acf71dfc8726b00a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:36:51.943384 18611 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:36:51.943472 18606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:36:51.944444 18606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:36:51.951403 18606 catalog_manager.cc:1347] Generated new cluster ID: 2c9a95e6596a476bae9610ae108d2890
I20260929 18:36:51.951499 18606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:36:51.969731 18606 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:36:51.970264 18606 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:36:51.983222 18606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4fae354286e47b8acf71dfc8726b00a: Generated new TSK 0
I20260929 18:36:51.984150 18606 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:36:51.987628 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 18518
2026-09-29T18:36:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1484 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:36:52 dist-test-slave-qcpv krb5kdc[18617](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:36:52 dist-test-slave-qcpv krb5kdc[18617](info): set up 2 sockets
Sep 29 18:36:52 dist-test-slave-qcpv krb5kdc[18617](info): commencing operation
krb5kdc: starting...
W20260929 18:36:53.111774 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:36:53 dist-test-slave-qcpv krb5kdc[18617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:36:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:36:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.207.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
I20260929 18:36:53.258958 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:35811
--webserver_interface=127.28.207.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:38375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:35811
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:53.376435 18630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:53.376760 18630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:53.376834 18630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:53.380962 18630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:36:53.381067 18630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:53.381103 18630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:53.381142 18630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:36:53.381173 18630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:36:53.386773 18630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:38375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:35811
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:35811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:53.387887 18630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:53.388851 18630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:53.397694 18638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:53.397715 18635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:53.397867 18636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:53.397938 18630 server_base.cc:1034] running on GCE node
I20260929 18:36:53.398700 18630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:53.399875 18630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:53.401046 18630 hybrid_clock.cc:648] HybridClock initialized: now 1790707013401037 us; error 45 us; skew 500 ppm
Sep 29 18:36:53 dist-test-slave-qcpv krb5kdc[18617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707013, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:53.404408 18630 init.cc:377] Logged in from keytab as kudu/127.28.207.126@KRBTEST.COM (short username kudu)
I20260929 18:36:53.405582 18630 webserver.cc:460] Webserver started at http://127.28.207.126:46493/ using document root <none> and password file <none>
I20260929 18:36:53.406188 18630 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:53.406270 18630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:53.406505 18630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:53.408706 18630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "5a74107406f64d5a8137e0ab16ae9798"
format_stamp: "Formatted at 2026-09-29 18:36:53 on dist-test-slave-qcpv"
server_key: "01ca59a9e548a7e2dcc1d66ee2a22339"
server_key_iv: "2a1ed6d7237d02d26b2509e0ad8ea5ee"
server_key_version: "encryptionkey@0"
I20260929 18:36:53.409301 18630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "5a74107406f64d5a8137e0ab16ae9798"
format_stamp: "Formatted at 2026-09-29 18:36:53 on dist-test-slave-qcpv"
server_key: "01ca59a9e548a7e2dcc1d66ee2a22339"
server_key_iv: "2a1ed6d7237d02d26b2509e0ad8ea5ee"
server_key_version: "encryptionkey@0"
I20260929 18:36:53.413902 18630 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 18:36:53.416923 18645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:53.418550 18630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 18:36:53.418684 18630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "5a74107406f64d5a8137e0ab16ae9798"
format_stamp: "Formatted at 2026-09-29 18:36:53 on dist-test-slave-qcpv"
server_key: "01ca59a9e548a7e2dcc1d66ee2a22339"
server_key_iv: "2a1ed6d7237d02d26b2509e0ad8ea5ee"
server_key_version: "encryptionkey@0"
I20260929 18:36:53.418917 18630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:53.437712 18630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:53.441331 18630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:53.441543 18630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:53.449532 18630 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:35811
I20260929 18:36:53.449640 18697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:35811 every 8 connection(s)
I20260929 18:36:53.450608 18630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:36:53.454620 18698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:53.459144 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 18630
I20260929 18:36:53.459324 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:36:53.459704 29501 external_mini_cluster.cc:1428] Setting key 2be07383cf628dc8f6ebfc44c8880913
I20260929 18:36:53.462304 18698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a74107406f64d5a8137e0ab16ae9798: Bootstrap starting.
I20260929 18:36:53.465379 18698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a74107406f64d5a8137e0ab16ae9798: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:53.466306 18698 log.cc:824] T 00000000000000000000000000000000 P 5a74107406f64d5a8137e0ab16ae9798: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:53.468701 18698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a74107406f64d5a8137e0ab16ae9798: No bootstrap required, opened a new log
Sep 29 18:36:53 dist-test-slave-qcpv krb5kdc[18617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:36:53.473996 18698 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a74107406f64d5a8137e0ab16ae9798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a74107406f64d5a8137e0ab16ae9798" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 35811 } }
I20260929 18:36:53.474363 18698 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a74107406f64d5a8137e0ab16ae9798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:53.474426 18698 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a74107406f64d5a8137e0ab16ae9798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a74107406f64d5a8137e0ab16ae9798, State: Initialized, Role: FOLLOWER
I20260929 18:36:53.475075 18698 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a74107406f64d5a8137e0ab16ae9798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a74107406f64d5a8137e0ab16ae9798" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 35811 } }
I20260929 18:36:53.475257 18698 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a74107406f64d5a8137e0ab16ae9798 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:36:53.475350 18698 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a74107406f64d5a8137e0ab16ae9798 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:36:53.475487 18698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a74107406f64d5a8137e0ab16ae9798 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:53.476639 18698 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a74107406f64d5a8137e0ab16ae9798 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a74107406f64d5a8137e0ab16ae9798" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 35811 } }
I20260929 18:36:53.477102 18698 leader_election.cc:302] T 00000000000000000000000000000000 P 5a74107406f64d5a8137e0ab16ae9798 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a74107406f64d5a8137e0ab16ae9798; no voters:
I20260929 18:36:53.477643 18698 leader_election.cc:288] T 00000000000000000000000000000000 P 5a74107406f64d5a8137e0ab16ae9798 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:36:53.477836 18703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a74107406f64d5a8137e0ab16ae9798 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:53.478241 18703 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a74107406f64d5a8137e0ab16ae9798 [term 1 LEADER]: Becoming Leader. State: Replica: 5a74107406f64d5a8137e0ab16ae9798, State: Running, Role: LEADER
I20260929 18:36:53.478672 18701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:53.461593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37541 (local address 127.28.207.126:35811)
0929 18:36:53.462262 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:53.462310 (+ 48us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:53.462361 (+ 51us) server_negotiation.cc:407] Connection header received
0929 18:36:53.463410 (+ 1049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:53.463433 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:53.463953 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:53.464375 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:53.465398 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:53.466651 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:53.467667 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:53.468222 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:53.468418 (+ 196us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:53.471726 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:53.471734 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:53.471762 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:53.471812 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:53.474189 (+ 2377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:53.475043 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:53.475051 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:53.475058 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:53.475360 (+ 302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:53.475818 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:53.475821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:53.475823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:53.476330 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:53.476609 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:53.477003 (+ 394us) server_negotiation.cc:299] Negotiation successful
0929 18:36:53.477417 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":191,"threads_started":1}
I20260929 18:36:53.478711 18703 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a74107406f64d5a8137e0ab16ae9798 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a74107406f64d5a8137e0ab16ae9798" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 35811 } }
I20260929 18:36:53.479346 18698 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a74107406f64d5a8137e0ab16ae9798 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:36:53.481619 18704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a74107406f64d5a8137e0ab16ae9798 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a74107406f64d5a8137e0ab16ae9798" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a74107406f64d5a8137e0ab16ae9798" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 35811 } } }
I20260929 18:36:53.481783 18704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a74107406f64d5a8137e0ab16ae9798 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:53.482478 18705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a74107406f64d5a8137e0ab16ae9798 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a74107406f64d5a8137e0ab16ae9798. Latest consensus state: current_term: 1 leader_uuid: "5a74107406f64d5a8137e0ab16ae9798" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a74107406f64d5a8137e0ab16ae9798" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 35811 } } }
I20260929 18:36:53.482692 18718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:36:53.482892 18705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a74107406f64d5a8137e0ab16ae9798 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:53.482973 29501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:36:53.486771 18718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:36:53.494386 18718 catalog_manager.cc:1347] Generated new cluster ID: 38b9796595b94305aa7610d6b664f24f
I20260929 18:36:53.494545 18718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:36:53.497206 18701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:53.485699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55745 (local address 127.28.207.126:35811)
0929 18:36:53.485836 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:53.485840 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:53.486094 (+ 254us) server_negotiation.cc:407] Connection header received
0929 18:36:53.486235 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:53.486238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:53.486297 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:53.486419 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:53.487348 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:53.488153 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:53.489402 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:53.489754 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:53.489875 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:53.492511 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:53.492514 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:53.492516 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:53.492543 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:53.494961 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:53.495615 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:53.495618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:53.495620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:53.495745 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:53.496122 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:53.496125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:53.496127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:53.496347 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:53.496482 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:53.496806 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 18:36:53.496917 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 18:36:53.509618 18718 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:36:53.511175 18718 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:36:53.520355 18718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a74107406f64d5a8137e0ab16ae9798: Generated new TSK 0
I20260929 18:36:53.521693 18718 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:36:53.527772 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 18630
W20260929 18:36:53.533819 18615 connection.cc:535] client connection to 127.28.207.126:35811 recv error: Network error: failed to read from TLS socket (remote: 127.28.207.126:35811): Connection reset by peer (error 104)
2026-09-29T18:36:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1540 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:36:53 dist-test-slave-qcpv krb5kdc[18729](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:36:53 dist-test-slave-qcpv krb5kdc[18729](info): set up 2 sockets
Sep 29 18:36:53 dist-test-slave-qcpv krb5kdc[18729](info): commencing operation
krb5kdc: starting...
W20260929 18:36:54.619498 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:36:54 dist-test-slave-qcpv krb5kdc[18729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:36:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:36:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.207.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
I20260929 18:36:54.766716 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:46789
--webserver_interface=127.28.207.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:45119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:46789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:54.884330 18742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:54.884738 18742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:54.884824 18742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:54.889123 18742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:36:54.889236 18742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:54.889274 18742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:54.889314 18742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:36:54.889350 18742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:36:54.895083 18742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:45119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:46789
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:46789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:54.896270 18742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:54.897363 18742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:54.906673 18750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:54.906688 18748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:54.907064 18742 server_base.cc:1034] running on GCE node
W20260929 18:36:54.906700 18747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:54.907785 18742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:54.909062 18742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:54.910319 18742 hybrid_clock.cc:648] HybridClock initialized: now 1790707014910274 us; error 76 us; skew 500 ppm
Sep 29 18:36:54 dist-test-slave-qcpv krb5kdc[18729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707014, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:54.913507 18742 init.cc:377] Logged in from keytab as kudu/127.28.207.126@KRBTEST.COM (short username kudu)
I20260929 18:36:54.914744 18742 webserver.cc:460] Webserver started at http://127.28.207.126:45563/ using document root <none> and password file <none>
I20260929 18:36:54.915373 18742 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:54.915477 18742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:54.915715 18742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:54.918043 18742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "d161c1fcd41240e38f9c48dc76956b76"
format_stamp: "Formatted at 2026-09-29 18:36:54 on dist-test-slave-qcpv"
server_key: "f4e28bdc2aab8f332c18af657d85c2de"
server_key_iv: "b685c43cb089157ec4368071526281d3"
server_key_version: "encryptionkey@0"
I20260929 18:36:54.918678 18742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "d161c1fcd41240e38f9c48dc76956b76"
format_stamp: "Formatted at 2026-09-29 18:36:54 on dist-test-slave-qcpv"
server_key: "f4e28bdc2aab8f332c18af657d85c2de"
server_key_iv: "b685c43cb089157ec4368071526281d3"
server_key_version: "encryptionkey@0"
I20260929 18:36:54.923211 18742 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:36:54.926402 18757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:54.928833 18742 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:36:54.929029 18742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "d161c1fcd41240e38f9c48dc76956b76"
format_stamp: "Formatted at 2026-09-29 18:36:54 on dist-test-slave-qcpv"
server_key: "f4e28bdc2aab8f332c18af657d85c2de"
server_key_iv: "b685c43cb089157ec4368071526281d3"
server_key_version: "encryptionkey@0"
I20260929 18:36:54.929176 18742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:54.949229 18742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:54.953262 18742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:54.953495 18742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:54.962773 18742 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:46789
I20260929 18:36:54.962985 18809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:46789 every 8 connection(s)
I20260929 18:36:54.964092 18742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:36:54.966498 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 18742
I20260929 18:36:54.966683 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:36:54.967125 29501 external_mini_cluster.cc:1428] Setting key dec8a1f60081a5190632854f57afe8f4
I20260929 18:36:54.968634 18810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:54.977456 18810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d161c1fcd41240e38f9c48dc76956b76: Bootstrap starting.
I20260929 18:36:54.979406 18813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:54.969221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59699 (local address 127.28.207.126:46789)
0929 18:36:54.969915 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:54.969935 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:54.969985 (+ 50us) server_negotiation.cc:407] Connection header received
0929 18:36:54.970754 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:54.970777 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:54.971430 (+ 653us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:54.971935 (+ 505us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:54.972472 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:54.973573 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:54.974566 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:54.975248 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:54.975442 (+ 194us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:54.976513 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:54.976519 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:54.976536 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:54.977039 (+ 503us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:54.977108 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:54.977140 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:54.977767 (+ 627us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:54.978109 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 18:36:54.978324 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":247,"threads_started":1}
I20260929 18:36:54.980947 18810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d161c1fcd41240e38f9c48dc76956b76: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:54.982111 18810 log.cc:824] T 00000000000000000000000000000000 P d161c1fcd41240e38f9c48dc76956b76: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:54.984565 29501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:36:54.986713 18810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d161c1fcd41240e38f9c48dc76956b76: No bootstrap required, opened a new log
I20260929 18:36:54.994515 18810 raft_consensus.cc:348] T 00000000000000000000000000000000 P d161c1fcd41240e38f9c48dc76956b76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d161c1fcd41240e38f9c48dc76956b76" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 46789 } }
I20260929 18:36:54.995021 18810 raft_consensus.cc:374] T 00000000000000000000000000000000 P d161c1fcd41240e38f9c48dc76956b76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:54.995169 18810 raft_consensus.cc:729] T 00000000000000000000000000000000 P d161c1fcd41240e38f9c48dc76956b76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d161c1fcd41240e38f9c48dc76956b76, State: Initialized, Role: FOLLOWER
I20260929 18:36:54.995929 18810 consensus_queue.cc:260] T 00000000000000000000000000000000 P d161c1fcd41240e38f9c48dc76956b76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d161c1fcd41240e38f9c48dc76956b76" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 46789 } }
I20260929 18:36:54.996135 18810 raft_consensus.cc:388] T 00000000000000000000000000000000 P d161c1fcd41240e38f9c48dc76956b76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:36:54.996245 18810 raft_consensus.cc:482] T 00000000000000000000000000000000 P d161c1fcd41240e38f9c48dc76956b76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:36:54.996376 18810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d161c1fcd41240e38f9c48dc76956b76 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:54.998219 18810 raft_consensus.cc:504] T 00000000000000000000000000000000 P d161c1fcd41240e38f9c48dc76956b76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d161c1fcd41240e38f9c48dc76956b76" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 46789 } }
I20260929 18:36:54.998463 18813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:54.992612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35933 (local address 127.28.207.126:46789)
0929 18:36:54.992768 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:54.992774 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:54.992793 (+ 19us) server_negotiation.cc:407] Connection header received
0929 18:36:54.992849 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:54.992856 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:54.992921 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:54.993047 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:54.993723 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:54.994494 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:54.995905 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:54.996214 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:54.996370 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:54.997657 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:54.997660 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:54.997663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:54.997680 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:54.997737 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:54.997751 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:54.997884 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:54.998131 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 18:36:54.998233 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 18:36:54.998827 18810 leader_election.cc:302] T 00000000000000000000000000000000 P d161c1fcd41240e38f9c48dc76956b76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d161c1fcd41240e38f9c48dc76956b76; no voters:
I20260929 18:36:54.999357 18810 leader_election.cc:288] T 00000000000000000000000000000000 P d161c1fcd41240e38f9c48dc76956b76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:36:54.999512 18820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d161c1fcd41240e38f9c48dc76956b76 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:54.999938 18820 raft_consensus.cc:686] T 00000000000000000000000000000000 P d161c1fcd41240e38f9c48dc76956b76 [term 1 LEADER]: Becoming Leader. State: Replica: d161c1fcd41240e38f9c48dc76956b76, State: Running, Role: LEADER
I20260929 18:36:55.000339 18820 consensus_queue.cc:237] T 00000000000000000000000000000000 P d161c1fcd41240e38f9c48dc76956b76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d161c1fcd41240e38f9c48dc76956b76" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 46789 } }
I20260929 18:36:55.001008 18810 sys_catalog.cc:564] T 00000000000000000000000000000000 P d161c1fcd41240e38f9c48dc76956b76 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:36:55.002828 18821 sys_catalog.cc:455] T 00000000000000000000000000000000 P d161c1fcd41240e38f9c48dc76956b76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d161c1fcd41240e38f9c48dc76956b76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d161c1fcd41240e38f9c48dc76956b76" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 46789 } } }
I20260929 18:36:55.002962 18822 sys_catalog.cc:455] T 00000000000000000000000000000000 P d161c1fcd41240e38f9c48dc76956b76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d161c1fcd41240e38f9c48dc76956b76. Latest consensus state: current_term: 1 leader_uuid: "d161c1fcd41240e38f9c48dc76956b76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d161c1fcd41240e38f9c48dc76956b76" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 46789 } } }
I20260929 18:36:55.003109 18821 sys_catalog.cc:458] T 00000000000000000000000000000000 P d161c1fcd41240e38f9c48dc76956b76 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:55.003134 18822 sys_catalog.cc:458] T 00000000000000000000000000000000 P d161c1fcd41240e38f9c48dc76956b76 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:55.003757 18831 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:36:55.007500 18831 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:36:55.016677 18831 catalog_manager.cc:1347] Generated new cluster ID: f850c2f27187412f81f351464f380b33
I20260929 18:36:55.016790 18831 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:36:55.042450 18831 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:36:55.044169 18831 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:36:55.055085 18831 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d161c1fcd41240e38f9c48dc76956b76: Generated new TSK 0
I20260929 18:36:55.056471 18831 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:36:55.064114 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 18742
W20260929 18:36:55.071689 18727 connection.cc:535] client connection to 127.28.207.126:46789 recv error: Network error: failed to read from TLS socket (remote: 127.28.207.126:46789): Connection reset by peer (error 104)
2026-09-29T18:36:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1538 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:36:55 dist-test-slave-qcpv krb5kdc[18841](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:36:55 dist-test-slave-qcpv krb5kdc[18841](info): set up 2 sockets
Sep 29 18:36:55 dist-test-slave-qcpv krb5kdc[18841](info): commencing operation
krb5kdc: starting...
W20260929 18:36:56.164076 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:36:56 dist-test-slave-qcpv krb5kdc[18841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:36:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:36:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.207.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
I20260929 18:36:56.307571 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:33255
--webserver_interface=127.28.207.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:33751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:33255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:56.446835 18854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:56.447263 18854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:56.447314 18854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:56.452376 18854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:36:56.452489 18854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:56.452512 18854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:56.452531 18854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:36:56.452548 18854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:36:56.458974 18854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:33751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:33255
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:33255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:56.460467 18854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:56.461746 18854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:56.473060 18859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:56.473060 18862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:56.473063 18860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:56.473453 18854 server_base.cc:1034] running on GCE node
I20260929 18:36:56.474404 18854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:56.476027 18854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:56.477515 18854 hybrid_clock.cc:648] HybridClock initialized: now 1790707016477492 us; error 44 us; skew 500 ppm
Sep 29 18:36:56 dist-test-slave-qcpv krb5kdc[18841](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707016, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:56.487180 18854 init.cc:377] Logged in from keytab as kudu/127.28.207.126@KRBTEST.COM (short username kudu)
I20260929 18:36:56.488780 18854 webserver.cc:460] Webserver started at http://127.28.207.126:43361/ using document root <none> and password file <none>
I20260929 18:36:56.489655 18854 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:56.489777 18854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:56.490063 18854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:56.493054 18854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "db487ecc0a1a4eb9a5c2ee0d5847ebcd"
format_stamp: "Formatted at 2026-09-29 18:36:56 on dist-test-slave-qcpv"
server_key: "bee5d967441090e597bd19419d850082"
server_key_iv: "3f917d45d2c9a3b1c03b0ccb422f2d4d"
server_key_version: "encryptionkey@0"
I20260929 18:36:56.494086 18854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "db487ecc0a1a4eb9a5c2ee0d5847ebcd"
format_stamp: "Formatted at 2026-09-29 18:36:56 on dist-test-slave-qcpv"
server_key: "bee5d967441090e597bd19419d850082"
server_key_iv: "3f917d45d2c9a3b1c03b0ccb422f2d4d"
server_key_version: "encryptionkey@0"
I20260929 18:36:56.499503 18854 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 18:36:56.503693 18869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:56.505784 18854 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:36:56.506006 18854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "db487ecc0a1a4eb9a5c2ee0d5847ebcd"
format_stamp: "Formatted at 2026-09-29 18:36:56 on dist-test-slave-qcpv"
server_key: "bee5d967441090e597bd19419d850082"
server_key_iv: "3f917d45d2c9a3b1c03b0ccb422f2d4d"
server_key_version: "encryptionkey@0"
I20260929 18:36:56.506181 18854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:56.511787 18854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:56.516814 18854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:56.517091 18854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:56.526419 18854 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:33255
I20260929 18:36:56.526647 18921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:33255 every 8 connection(s)
I20260929 18:36:56.527891 18854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:36:56.531183 18922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:56.536923 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 18854
I20260929 18:36:56.537081 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:36:56.537294 18922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db487ecc0a1a4eb9a5c2ee0d5847ebcd: Bootstrap starting.
I20260929 18:36:56.537424 29501 external_mini_cluster.cc:1428] Setting key 94cff34d6e3abacfbd97336bb7af2aa8
I20260929 18:36:56.540108 18922 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db487ecc0a1a4eb9a5c2ee0d5847ebcd: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:56.540918 18922 log.cc:824] T 00000000000000000000000000000000 P db487ecc0a1a4eb9a5c2ee0d5847ebcd: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:56.543860 18922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db487ecc0a1a4eb9a5c2ee0d5847ebcd: No bootstrap required, opened a new log
I20260929 18:36:56.544741 18925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:56.539323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40843 (local address 127.28.207.126:33255)
0929 18:36:56.539933 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:56.539960 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:56.540019 (+ 59us) server_negotiation.cc:407] Connection header received
0929 18:36:56.541080 (+ 1061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:56.541108 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:56.541651 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:56.542136 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:56.543201 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:56.543207 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:56.543220 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:56.543231 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:56.543296 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:56.543318 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:56.543459 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:56.543543 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 18:36:56.543749 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":184,"threads_started":1}
I20260929 18:36:56.548673 29501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:36:56.548794 18922 raft_consensus.cc:348] T 00000000000000000000000000000000 P db487ecc0a1a4eb9a5c2ee0d5847ebcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db487ecc0a1a4eb9a5c2ee0d5847ebcd" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33255 } }
I20260929 18:36:56.549109 18922 raft_consensus.cc:374] T 00000000000000000000000000000000 P db487ecc0a1a4eb9a5c2ee0d5847ebcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:56.549196 18922 raft_consensus.cc:729] T 00000000000000000000000000000000 P db487ecc0a1a4eb9a5c2ee0d5847ebcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db487ecc0a1a4eb9a5c2ee0d5847ebcd, State: Initialized, Role: FOLLOWER
I20260929 18:36:56.549639 18922 consensus_queue.cc:260] T 00000000000000000000000000000000 P db487ecc0a1a4eb9a5c2ee0d5847ebcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db487ecc0a1a4eb9a5c2ee0d5847ebcd" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33255 } }
I20260929 18:36:56.549762 18922 raft_consensus.cc:388] T 00000000000000000000000000000000 P db487ecc0a1a4eb9a5c2ee0d5847ebcd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:36:56.549844 18922 raft_consensus.cc:482] T 00000000000000000000000000000000 P db487ecc0a1a4eb9a5c2ee0d5847ebcd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:36:56.549960 18922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db487ecc0a1a4eb9a5c2ee0d5847ebcd [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:56.551126 18922 raft_consensus.cc:504] T 00000000000000000000000000000000 P db487ecc0a1a4eb9a5c2ee0d5847ebcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db487ecc0a1a4eb9a5c2ee0d5847ebcd" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33255 } }
I20260929 18:36:56.551517 18922 leader_election.cc:302] T 00000000000000000000000000000000 P db487ecc0a1a4eb9a5c2ee0d5847ebcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db487ecc0a1a4eb9a5c2ee0d5847ebcd; no voters:
I20260929 18:36:56.551882 18922 leader_election.cc:288] T 00000000000000000000000000000000 P db487ecc0a1a4eb9a5c2ee0d5847ebcd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:36:56.552119 18931 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db487ecc0a1a4eb9a5c2ee0d5847ebcd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:56.552376 18931 raft_consensus.cc:686] T 00000000000000000000000000000000 P db487ecc0a1a4eb9a5c2ee0d5847ebcd [term 1 LEADER]: Becoming Leader. State: Replica: db487ecc0a1a4eb9a5c2ee0d5847ebcd, State: Running, Role: LEADER
I20260929 18:36:56.552830 18931 consensus_queue.cc:237] T 00000000000000000000000000000000 P db487ecc0a1a4eb9a5c2ee0d5847ebcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db487ecc0a1a4eb9a5c2ee0d5847ebcd" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33255 } }
I20260929 18:36:56.553524 18922 sys_catalog.cc:564] T 00000000000000000000000000000000 P db487ecc0a1a4eb9a5c2ee0d5847ebcd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:36:56.553956 18925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:56.551873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40397 (local address 127.28.207.126:33255)
0929 18:36:56.552000 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:56.552007 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:56.552262 (+ 255us) server_negotiation.cc:407] Connection header received
0929 18:36:56.552460 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:56.552463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:56.552505 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:56.552555 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:56.553460 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:56.553464 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:56.553467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:56.553479 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:56.553507 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:56.553520 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:56.553585 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:56.553761 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 18:36:56.553805 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 18:36:56.555111 18931 sys_catalog.cc:455] T 00000000000000000000000000000000 P db487ecc0a1a4eb9a5c2ee0d5847ebcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader db487ecc0a1a4eb9a5c2ee0d5847ebcd. Latest consensus state: current_term: 1 leader_uuid: "db487ecc0a1a4eb9a5c2ee0d5847ebcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db487ecc0a1a4eb9a5c2ee0d5847ebcd" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33255 } } }
I20260929 18:36:56.555315 18931 sys_catalog.cc:458] T 00000000000000000000000000000000 P db487ecc0a1a4eb9a5c2ee0d5847ebcd [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:56.557319 18933 sys_catalog.cc:455] T 00000000000000000000000000000000 P db487ecc0a1a4eb9a5c2ee0d5847ebcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db487ecc0a1a4eb9a5c2ee0d5847ebcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db487ecc0a1a4eb9a5c2ee0d5847ebcd" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 33255 } } }
I20260929 18:36:56.557477 18933 sys_catalog.cc:458] T 00000000000000000000000000000000 P db487ecc0a1a4eb9a5c2ee0d5847ebcd [sys.catalog]: This master's current role is: LEADER
W20260929 18:36:56.559691 18947 catalog_manager.cc:1558] T 00000000000000000000000000000000 P db487ecc0a1a4eb9a5c2ee0d5847ebcd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:36:56.559805 18947 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:36:56.559998 18944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:36:56.560992 18944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:36:56.568670 18944 catalog_manager.cc:1347] Generated new cluster ID: 0ef1ed8547554043ad06eae16f868f2a
I20260929 18:36:56.568778 18944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:36:56.596516 18944 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:36:56.597002 18944 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:36:56.603789 18944 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db487ecc0a1a4eb9a5c2ee0d5847ebcd: Generated new TSK 0
I20260929 18:36:56.604723 18944 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:36:56.613797 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 18854
2026-09-29T18:36:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1547 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:36:56 dist-test-slave-qcpv krb5kdc[18953](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:36:56 dist-test-slave-qcpv krb5kdc[18953](info): set up 2 sockets
Sep 29 18:36:56 dist-test-slave-qcpv krb5kdc[18953](info): commencing operation
krb5kdc: starting...
W20260929 18:36:57.703540 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:36:57 dist-test-slave-qcpv krb5kdc[18953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:36:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:36:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.207.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
I20260929 18:36:57.847987 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:43211
--webserver_interface=127.28.207.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:35901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:43211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:57.978036 18966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:57.978415 18966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:57.978515 18966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:57.983963 18966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:36:57.984110 18966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:57.984149 18966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:57.984181 18966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:36:57.984239 18966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:36:57.991151 18966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:35901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:43211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:43211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:57.992512 18966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:57.993705 18966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:58.006163 18974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:58.006163 18971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:58.006297 18972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:58.006744 18966 server_base.cc:1034] running on GCE node
I20260929 18:36:58.007471 18966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:58.008826 18966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:58.010185 18966 hybrid_clock.cc:648] HybridClock initialized: now 1790707018010166 us; error 39 us; skew 500 ppm
Sep 29 18:36:58 dist-test-slave-qcpv krb5kdc[18953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707018, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:58.014780 18966 init.cc:377] Logged in from keytab as kudu/127.28.207.126@KRBTEST.COM (short username kudu)
I20260929 18:36:58.016177 18966 webserver.cc:460] Webserver started at http://127.28.207.126:38991/ using document root <none> and password file <none>
I20260929 18:36:58.016834 18966 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:58.016937 18966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:58.017882 18966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:58.020526 18966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "5ee98c18bec6406da788ade5e40f139f"
format_stamp: "Formatted at 2026-09-29 18:36:58 on dist-test-slave-qcpv"
server_key: "90902dcd7f5c9a8d93a74b8c9dc894b8"
server_key_iv: "40eb2c6baa372a135f252bb1ce3becc6"
server_key_version: "encryptionkey@0"
I20260929 18:36:58.021184 18966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "5ee98c18bec6406da788ade5e40f139f"
format_stamp: "Formatted at 2026-09-29 18:36:58 on dist-test-slave-qcpv"
server_key: "90902dcd7f5c9a8d93a74b8c9dc894b8"
server_key_iv: "40eb2c6baa372a135f252bb1ce3becc6"
server_key_version: "encryptionkey@0"
I20260929 18:36:58.026264 18966 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 18:36:58.029543 18981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:58.032227 18966 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:36:58.032485 18966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "5ee98c18bec6406da788ade5e40f139f"
format_stamp: "Formatted at 2026-09-29 18:36:58 on dist-test-slave-qcpv"
server_key: "90902dcd7f5c9a8d93a74b8c9dc894b8"
server_key_iv: "40eb2c6baa372a135f252bb1ce3becc6"
server_key_version: "encryptionkey@0"
I20260929 18:36:58.032666 18966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:58.080713 18966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:58.084738 18966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:58.085047 18966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:58.094101 18966 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:43211
I20260929 18:36:58.094187 19033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:43211 every 8 connection(s)
I20260929 18:36:58.095381 18966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:36:58.099268 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 18966
I20260929 18:36:58.099432 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:36:58.099671 19034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:58.099818 29501 external_mini_cluster.cc:1428] Setting key baba07e75576b0a7b98d61a6b7e2be92
I20260929 18:36:58.109218 19034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ee98c18bec6406da788ade5e40f139f: Bootstrap starting.
Sep 29 18:36:58 dist-test-slave-qcpv krb5kdc[18953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:36:58.112455 19034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ee98c18bec6406da788ade5e40f139f: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:58.113421 19034 log.cc:824] T 00000000000000000000000000000000 P 5ee98c18bec6406da788ade5e40f139f: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:58.116406 19034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ee98c18bec6406da788ade5e40f139f: No bootstrap required, opened a new log
I20260929 18:36:58.119251 19037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:58.101893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33367 (local address 127.28.207.126:43211)
0929 18:36:58.102576 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:58.102598 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:58.102775 (+ 177us) server_negotiation.cc:407] Connection header received
0929 18:36:58.104036 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:58.104060 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:58.104501 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:58.104882 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:58.106011 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:58.107299 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:58.108684 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:58.109321 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:58.109554 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:58.113136 (+ 3582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:58.113143 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:58.113169 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:58.113207 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:58.115737 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:58.116395 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:58.116406 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:58.116413 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:58.116586 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:58.116901 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:58.116905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:58.116907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:58.117409 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:58.117653 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:58.117956 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 18:36:58.118312 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":200,"threads_started":1}
I20260929 18:36:58.121210 19034 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ee98c18bec6406da788ade5e40f139f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee98c18bec6406da788ade5e40f139f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43211 } }
I20260929 18:36:58.121656 19034 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ee98c18bec6406da788ade5e40f139f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:58.121739 19034 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ee98c18bec6406da788ade5e40f139f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee98c18bec6406da788ade5e40f139f, State: Initialized, Role: FOLLOWER
I20260929 18:36:58.122316 19034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ee98c18bec6406da788ade5e40f139f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee98c18bec6406da788ade5e40f139f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43211 } }
I20260929 18:36:58.122509 19034 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ee98c18bec6406da788ade5e40f139f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:36:58.122598 19034 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ee98c18bec6406da788ade5e40f139f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:36:58.122733 19034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ee98c18bec6406da788ade5e40f139f [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:58.122839 29501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:36:58.124138 19034 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ee98c18bec6406da788ade5e40f139f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee98c18bec6406da788ade5e40f139f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43211 } }
I20260929 18:36:58.124640 19034 leader_election.cc:302] T 00000000000000000000000000000000 P 5ee98c18bec6406da788ade5e40f139f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ee98c18bec6406da788ade5e40f139f; no voters:
I20260929 18:36:58.125119 19034 leader_election.cc:288] T 00000000000000000000000000000000 P 5ee98c18bec6406da788ade5e40f139f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:36:58.125348 19043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ee98c18bec6406da788ade5e40f139f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:58.125910 19043 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ee98c18bec6406da788ade5e40f139f [term 1 LEADER]: Becoming Leader. State: Replica: 5ee98c18bec6406da788ade5e40f139f, State: Running, Role: LEADER
I20260929 18:36:58.126423 19043 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ee98c18bec6406da788ade5e40f139f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee98c18bec6406da788ade5e40f139f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43211 } }
I20260929 18:36:58.127171 19034 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ee98c18bec6406da788ade5e40f139f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:36:58.128387 19045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ee98c18bec6406da788ade5e40f139f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ee98c18bec6406da788ade5e40f139f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee98c18bec6406da788ade5e40f139f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43211 } } }
I20260929 18:36:58.128551 19045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ee98c18bec6406da788ade5e40f139f [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:58.129022 19046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ee98c18bec6406da788ade5e40f139f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ee98c18bec6406da788ade5e40f139f. Latest consensus state: current_term: 1 leader_uuid: "5ee98c18bec6406da788ade5e40f139f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee98c18bec6406da788ade5e40f139f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43211 } } }
I20260929 18:36:58.129187 19046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ee98c18bec6406da788ade5e40f139f [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:58.129518 19053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:36:58.136271 19053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:36:58.137552 19037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:58.125392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44621 (local address 127.28.207.126:43211)
0929 18:36:58.125538 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:58.125544 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:58.125847 (+ 303us) server_negotiation.cc:407] Connection header received
0929 18:36:58.126044 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:58.126049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:58.126112 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:58.126172 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:58.127436 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:58.128344 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:58.129762 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:58.130112 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:58.130242 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:58.132480 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:58.132483 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:58.132486 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:36:58.132516 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:58.134534 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:58.135280 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:58.135287 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:58.135290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:58.135384 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:36:58.135749 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:36:58.135752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:36:58.135757 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:36:58.136046 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:58.136935 (+ 889us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:58.137026 (+ 91us) server_negotiation.cc:299] Negotiation successful
0929 18:36:58.137173 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 18:36:58.146526 19053 catalog_manager.cc:1347] Generated new cluster ID: 7a45ab3d4dab45de9a56c7a6aae3405d
I20260929 18:36:58.146642 19053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:36:58.161653 19053 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:36:58.162782 19053 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:36:58.179129 19053 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ee98c18bec6406da788ade5e40f139f: Generated new TSK 0
I20260929 18:36:58.180070 19053 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:36:58.193048 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 18966
2026-09-29T18:36:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1580 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:36:58 dist-test-slave-qcpv krb5kdc[19065](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:36:58 dist-test-slave-qcpv krb5kdc[19065](info): set up 2 sockets
Sep 29 18:36:58 dist-test-slave-qcpv krb5kdc[19065](info): commencing operation
krb5kdc: starting...
W20260929 18:36:59.307502 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:36:59 dist-test-slave-qcpv krb5kdc[19065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:36:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:36:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.207.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
I20260929 18:36:59.450691 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:39909
--webserver_interface=127.28.207.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:38001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:39909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:36:59.568768 19078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:36:59.569089 19078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:36:59.569135 19078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:36:59.573283 19078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:36:59.573362 19078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:36:59.573382 19078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:36:59.573401 19078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:36:59.573417 19078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:36:59.578933 19078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:38001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:39909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:39909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:36:59.580013 19078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:36:59.580988 19078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:36:59.590551 19086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:59.590551 19084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:36:59.590689 19083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:36:59.591370 19078 server_base.cc:1034] running on GCE node
I20260929 18:36:59.592211 19078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:36:59.593894 19078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:36:59.595381 19078 hybrid_clock.cc:648] HybridClock initialized: now 1790707019595333 us; error 66 us; skew 500 ppm
Sep 29 18:36:59 dist-test-slave-qcpv krb5kdc[19065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707019, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:36:59.599153 19078 init.cc:377] Logged in from keytab as kudu/127.28.207.126@KRBTEST.COM (short username kudu)
I20260929 18:36:59.600317 19078 webserver.cc:460] Webserver started at http://127.28.207.126:33535/ using document root <none> and password file <none>
I20260929 18:36:59.600983 19078 fs_manager.cc:359] Metadata directory not provided
I20260929 18:36:59.601050 19078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:36:59.601269 19078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:36:59.603920 19078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "53c1427d55704c18bbd055d9e75c0b65"
format_stamp: "Formatted at 2026-09-29 18:36:59 on dist-test-slave-qcpv"
server_key: "3c8cf5d1f26548be6aa79f16692f6642"
server_key_iv: "a4b9327187166702022d023ca8db3fcd"
server_key_version: "encryptionkey@0"
I20260929 18:36:59.604548 19078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "53c1427d55704c18bbd055d9e75c0b65"
format_stamp: "Formatted at 2026-09-29 18:36:59 on dist-test-slave-qcpv"
server_key: "3c8cf5d1f26548be6aa79f16692f6642"
server_key_iv: "a4b9327187166702022d023ca8db3fcd"
server_key_version: "encryptionkey@0"
I20260929 18:36:59.609472 19078 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 18:36:59.612995 19093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:36:59.614850 19078 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 18:36:59.615139 19078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "53c1427d55704c18bbd055d9e75c0b65"
format_stamp: "Formatted at 2026-09-29 18:36:59 on dist-test-slave-qcpv"
server_key: "3c8cf5d1f26548be6aa79f16692f6642"
server_key_iv: "a4b9327187166702022d023ca8db3fcd"
server_key_version: "encryptionkey@0"
I20260929 18:36:59.615304 19078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:36:59.652439 19078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:36:59.656854 19078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:36:59.657124 19078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:36:59.666582 19078 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:39909
I20260929 18:36:59.666625 19145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:39909 every 8 connection(s)
I20260929 18:36:59.668128 19078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:36:59.670610 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 19078
I20260929 18:36:59.670740 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:36:59.671104 29501 external_mini_cluster.cc:1428] Setting key 16a6dffbd84f6294408db53c43054c68
I20260929 18:36:59.671864 19146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:36:59.679708 19146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53c1427d55704c18bbd055d9e75c0b65: Bootstrap starting.
I20260929 18:36:59.682760 19146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53c1427d55704c18bbd055d9e75c0b65: Neither blocks nor log segments found. Creating new log.
I20260929 18:36:59.682968 19149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:59.672920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58625 (local address 127.28.207.126:39909)
0929 18:36:59.673584 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:59.673605 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:59.673657 (+ 52us) server_negotiation.cc:407] Connection header received
0929 18:36:59.674515 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:59.674537 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:59.675111 (+ 574us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:59.675530 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:59.676044 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:59.677176 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:59.678195 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:59.678755 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:59.679004 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:59.680208 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:59.680214 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:59.680231 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:59.680756 (+ 525us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:59.680831 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:59.680858 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:59.680993 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:59.681128 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 18:36:59.681420 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":230,"threads_started":1}
I20260929 18:36:59.683848 19146 log.cc:824] T 00000000000000000000000000000000 P 53c1427d55704c18bbd055d9e75c0b65: Log is configured to *not* fsync() on all Append() calls
I20260929 18:36:59.686458 19146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53c1427d55704c18bbd055d9e75c0b65: No bootstrap required, opened a new log
I20260929 18:36:59.687305 29501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:36:59.690749 19146 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53c1427d55704c18bbd055d9e75c0b65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c1427d55704c18bbd055d9e75c0b65" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 39909 } }
I20260929 18:36:59.691147 19146 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53c1427d55704c18bbd055d9e75c0b65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:36:59.691222 19146 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53c1427d55704c18bbd055d9e75c0b65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53c1427d55704c18bbd055d9e75c0b65, State: Initialized, Role: FOLLOWER
I20260929 18:36:59.691751 19146 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53c1427d55704c18bbd055d9e75c0b65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c1427d55704c18bbd055d9e75c0b65" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 39909 } }
I20260929 18:36:59.691913 19146 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53c1427d55704c18bbd055d9e75c0b65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:36:59.691973 19146 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53c1427d55704c18bbd055d9e75c0b65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:36:59.692077 19146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53c1427d55704c18bbd055d9e75c0b65 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:36:59.693158 19146 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53c1427d55704c18bbd055d9e75c0b65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c1427d55704c18bbd055d9e75c0b65" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 39909 } }
I20260929 18:36:59.693569 19146 leader_election.cc:302] T 00000000000000000000000000000000 P 53c1427d55704c18bbd055d9e75c0b65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53c1427d55704c18bbd055d9e75c0b65; no voters:
I20260929 18:36:59.693971 19146 leader_election.cc:288] T 00000000000000000000000000000000 P 53c1427d55704c18bbd055d9e75c0b65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:36:59.694811 19157 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53c1427d55704c18bbd055d9e75c0b65 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:36:59.695096 19157 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53c1427d55704c18bbd055d9e75c0b65 [term 1 LEADER]: Becoming Leader. State: Replica: 53c1427d55704c18bbd055d9e75c0b65, State: Running, Role: LEADER
I20260929 18:36:59.695403 19149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:36:59.689907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52053 (local address 127.28.207.126:39909)
0929 18:36:59.690092 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 18:36:59.690097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:36:59.690371 (+ 274us) server_negotiation.cc:407] Connection header received
0929 18:36:59.690491 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:36:59.690494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:36:59.690560 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:36:59.690678 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:36:59.691150 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:59.691931 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:59.692961 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:36:59.693301 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:36:59.693410 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:36:59.694676 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:36:59.694680 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:36:59.694683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:36:59.694697 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:36:59.694727 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:36:59.694750 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:36:59.694828 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 18:36:59.695214 (+ 386us) server_negotiation.cc:299] Negotiation successful
0929 18:36:59.695273 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 18:36:59.695603 19146 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53c1427d55704c18bbd055d9e75c0b65 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:36:59.695827 19157 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53c1427d55704c18bbd055d9e75c0b65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c1427d55704c18bbd055d9e75c0b65" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 39909 } }
I20260929 18:36:59.698915 19156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53c1427d55704c18bbd055d9e75c0b65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53c1427d55704c18bbd055d9e75c0b65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c1427d55704c18bbd055d9e75c0b65" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 39909 } } }
I20260929 18:36:59.698807 19158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53c1427d55704c18bbd055d9e75c0b65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53c1427d55704c18bbd055d9e75c0b65. Latest consensus state: current_term: 1 leader_uuid: "53c1427d55704c18bbd055d9e75c0b65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c1427d55704c18bbd055d9e75c0b65" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 39909 } } }
I20260929 18:36:59.699021 19158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53c1427d55704c18bbd055d9e75c0b65 [sys.catalog]: This master's current role is: LEADER
I20260929 18:36:59.699021 19156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53c1427d55704c18bbd055d9e75c0b65 [sys.catalog]: This master's current role is: LEADER
W20260929 18:36:59.702553 19170 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 53c1427d55704c18bbd055d9e75c0b65: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:36:59.702682 19170 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:36:59.702754 19171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:36:59.703732 19171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:36:59.711349 19171 catalog_manager.cc:1347] Generated new cluster ID: 5c31fc3bf437494dac397b2432401663
I20260929 18:36:59.711474 19171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:36:59.732234 19171 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:36:59.733316 19171 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:36:59.742640 19171 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53c1427d55704c18bbd055d9e75c0b65: Generated new TSK 0
I20260929 18:36:59.743739 19171 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:36:59.755369 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 19078
2026-09-29T18:36:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1559 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:36:59 dist-test-slave-qcpv krb5kdc[19177](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:36:59 dist-test-slave-qcpv krb5kdc[19177](info): set up 2 sockets
Sep 29 18:36:59 dist-test-slave-qcpv krb5kdc[19177](info): commencing operation
krb5kdc: starting...
W20260929 18:37:00.839552 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:37:00 dist-test-slave-qcpv krb5kdc[19177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:37:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:37:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.207.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
I20260929 18:37:00.987999 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:36563
--webserver_interface=127.28.207.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:40739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:36563
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:01.107844 19190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:01.108222 19190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:01.108317 19190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:01.112578 19190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:37:01.112692 19190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:01.112731 19190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:01.112813 19190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:37:01.112849 19190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:37:01.120074 19190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:40739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:36563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:36563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:01.121312 19190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:01.122390 19190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:01.132498 19196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:01.132511 19195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:01.132715 19190 server_base.cc:1034] running on GCE node
W20260929 18:37:01.132511 19198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:01.133998 19190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:01.135716 19190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:01.136967 19190 hybrid_clock.cc:648] HybridClock initialized: now 1790707021136928 us; error 60 us; skew 500 ppm
Sep 29 18:37:01 dist-test-slave-qcpv krb5kdc[19177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707021, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:01.140913 19190 init.cc:377] Logged in from keytab as kudu/127.28.207.126@KRBTEST.COM (short username kudu)
I20260929 18:37:01.142194 19190 webserver.cc:460] Webserver started at http://127.28.207.126:37759/ using document root <none> and password file <none>
I20260929 18:37:01.143003 19190 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:01.143110 19190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:01.143393 19190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:01.146039 19190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "0dd17a2b86fe4525836c70713aca29e7"
format_stamp: "Formatted at 2026-09-29 18:37:01 on dist-test-slave-qcpv"
server_key: "224efac9062e10a1d34bc24c952039f4"
server_key_iv: "6e5e22ec1834fcb0383775de902c4961"
server_key_version: "encryptionkey@0"
I20260929 18:37:01.146719 19190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "0dd17a2b86fe4525836c70713aca29e7"
format_stamp: "Formatted at 2026-09-29 18:37:01 on dist-test-slave-qcpv"
server_key: "224efac9062e10a1d34bc24c952039f4"
server_key_iv: "6e5e22ec1834fcb0383775de902c4961"
server_key_version: "encryptionkey@0"
I20260929 18:37:01.151774 19190 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 18:37:01.155655 19205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:01.157765 19190 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:37:01.157991 19190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "0dd17a2b86fe4525836c70713aca29e7"
format_stamp: "Formatted at 2026-09-29 18:37:01 on dist-test-slave-qcpv"
server_key: "224efac9062e10a1d34bc24c952039f4"
server_key_iv: "6e5e22ec1834fcb0383775de902c4961"
server_key_version: "encryptionkey@0"
I20260929 18:37:01.158159 19190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:01.165145 19190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:01.169529 19190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:01.169888 19190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:01.179958 19190 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:36563
I20260929 18:37:01.179978 19257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:36563 every 8 connection(s)
I20260929 18:37:01.181200 19190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:37:01.184548 19258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:01.187110 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 19190
I20260929 18:37:01.187304 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:37:01.187655 29501 external_mini_cluster.cc:1428] Setting key 0864d0e32c043a8bf961e866bf0a13de
I20260929 18:37:01.191636 19258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dd17a2b86fe4525836c70713aca29e7: Bootstrap starting.
I20260929 18:37:01.194468 19261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:01.189605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34859 (local address 127.28.207.126:36563)
0929 18:37:01.190176 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:01.190197 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:01.190247 (+ 50us) server_negotiation.cc:407] Connection header received
0929 18:37:01.191171 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:01.191196 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:01.191598 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:01.192082 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:01.193004 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:01.193010 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:01.193023 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:37:01.193033 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:01.193095 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:37:01.193118 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:01.193294 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:01.193375 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 18:37:01.193565 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":138,"threads_started":1}
I20260929 18:37:01.194535 19258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0dd17a2b86fe4525836c70713aca29e7: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:01.195508 19258 log.cc:824] T 00000000000000000000000000000000 P 0dd17a2b86fe4525836c70713aca29e7: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:01.198081 19258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dd17a2b86fe4525836c70713aca29e7: No bootstrap required, opened a new log
I20260929 18:37:01.198133 29501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:37:01.203297 19261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:01.201219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40943 (local address 127.28.207.126:36563)
0929 18:37:01.201374 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:01.201379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:01.201511 (+ 132us) server_negotiation.cc:407] Connection header received
0929 18:37:01.201700 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:01.201704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:01.201753 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:01.201856 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:01.202751 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:01.202755 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:01.202758 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 18:37:01.202777 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:01.202805 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 18:37:01.202816 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:01.202986 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:01.203078 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 18:37:01.203127 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 18:37:01.203284 19258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0dd17a2b86fe4525836c70713aca29e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dd17a2b86fe4525836c70713aca29e7" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 36563 } }
I20260929 18:37:01.203688 19258 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0dd17a2b86fe4525836c70713aca29e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 18:37:01.203806 19258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0dd17a2b86fe4525836c70713aca29e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dd17a2b86fe4525836c70713aca29e7, State: Initialized, Role: FOLLOWER
I20260929 18:37:01.204242 19258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0dd17a2b86fe4525836c70713aca29e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dd17a2b86fe4525836c70713aca29e7" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 36563 } }
I20260929 18:37:01.204355 19258 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0dd17a2b86fe4525836c70713aca29e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 18:37:01.204396 19258 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0dd17a2b86fe4525836c70713aca29e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 18:37:01.204460 19258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0dd17a2b86fe4525836c70713aca29e7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:01.205542 19258 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0dd17a2b86fe4525836c70713aca29e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dd17a2b86fe4525836c70713aca29e7" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 36563 } }
I20260929 18:37:01.205920 19258 leader_election.cc:302] T 00000000000000000000000000000000 P 0dd17a2b86fe4525836c70713aca29e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0dd17a2b86fe4525836c70713aca29e7; no voters:
I20260929 18:37:01.206274 19258 leader_election.cc:288] T 00000000000000000000000000000000 P 0dd17a2b86fe4525836c70713aca29e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 18:37:01.206506 19268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0dd17a2b86fe4525836c70713aca29e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:37:01.206828 19268 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0dd17a2b86fe4525836c70713aca29e7 [term 1 LEADER]: Becoming Leader. State: Replica: 0dd17a2b86fe4525836c70713aca29e7, State: Running, Role: LEADER
I20260929 18:37:01.207314 19268 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0dd17a2b86fe4525836c70713aca29e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dd17a2b86fe4525836c70713aca29e7" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 36563 } }
I20260929 18:37:01.207701 19258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0dd17a2b86fe4525836c70713aca29e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:37:01.209594 19269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dd17a2b86fe4525836c70713aca29e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0dd17a2b86fe4525836c70713aca29e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dd17a2b86fe4525836c70713aca29e7" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 36563 } } }
I20260929 18:37:01.209789 19269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dd17a2b86fe4525836c70713aca29e7 [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:01.210342 19270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dd17a2b86fe4525836c70713aca29e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0dd17a2b86fe4525836c70713aca29e7. Latest consensus state: current_term: 1 leader_uuid: "0dd17a2b86fe4525836c70713aca29e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dd17a2b86fe4525836c70713aca29e7" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 36563 } } }
I20260929 18:37:01.210703 19270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dd17a2b86fe4525836c70713aca29e7 [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:01.211555 19282 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:37:01.214329 19282 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:37:01.222270 19282 catalog_manager.cc:1347] Generated new cluster ID: 5cfc0c01b3b646939d5fc17befd7b41b
I20260929 18:37:01.222393 19282 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:37:01.233878 19282 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:37:01.234293 19282 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:37:01.242439 19282 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0dd17a2b86fe4525836c70713aca29e7: Generated new TSK 0
I20260929 18:37:01.243335 19282 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 18:37:01.249325 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 19190
2026-09-29T18:37:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1494 ms)
[----------] 18 tests from AuthTokenIssuingTest (27481 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:37:01 dist-test-slave-qcpv krb5kdc[19289](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:37:01 dist-test-slave-qcpv krb5kdc[19289](info): set up 2 sockets
Sep 29 18:37:01 dist-test-slave-qcpv krb5kdc[19289](info): commencing operation
krb5kdc: starting...
W20260929 18:37:02.355437 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:37:02 dist-test-slave-qcpv krb5kdc[19289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:37:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:37:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.207.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
I20260929 18:37:02.508282 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:43313
--webserver_interface=127.28.207.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:45399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:43313,127.28.207.125:43445,127.28.207.124:39019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:02.628181 19302 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 18:37:02.628597 19302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:02.628654 19302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:02.628764 19302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:02.633111 19302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:37:02.633221 19302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:02.633262 19302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:02.633304 19302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:37:02.633344 19302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:37:02.639519 19302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:45399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:43313,127.28.207.125:43445,127.28.207.124:39019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:43313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:02.640746 19302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:02.641795 19302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:02.651335 19307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:02.651561 19302 server_base.cc:1034] running on GCE node
W20260929 18:37:02.651335 19310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:02.651401 19308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:02.652638 19302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:02.654244 19302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:02.655494 19302 hybrid_clock.cc:648] HybridClock initialized: now 1790707022655468 us; error 75 us; skew 500 ppm
Sep 29 18:37:02 dist-test-slave-qcpv krb5kdc[19289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707022, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:02.666778 19302 init.cc:377] Logged in from keytab as kudu/127.28.207.126@KRBTEST.COM (short username kudu)
I20260929 18:37:02.668190 19302 webserver.cc:460] Webserver started at http://127.28.207.126:41377/ using document root <none> and password file <none>
I20260929 18:37:02.668946 19302 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:02.669050 19302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:02.669341 19302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:02.672127 19302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "3b1ad8d91f7945f58403f8f6898f8c7f"
format_stamp: "Formatted at 2026-09-29 18:37:02 on dist-test-slave-qcpv"
server_key: "eaae44cf3092b3fbba743593d32c60c1"
server_key_iv: "ce8a2358d14a5e3161741a9bbe9ef97a"
server_key_version: "encryptionkey@0"
I20260929 18:37:02.672900 19302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "3b1ad8d91f7945f58403f8f6898f8c7f"
format_stamp: "Formatted at 2026-09-29 18:37:02 on dist-test-slave-qcpv"
server_key: "eaae44cf3092b3fbba743593d32c60c1"
server_key_iv: "ce8a2358d14a5e3161741a9bbe9ef97a"
server_key_version: "encryptionkey@0"
I20260929 18:37:02.678102 19302 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 18:37:02.681663 19317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:02.683876 19302 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:37:02.684085 19302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "3b1ad8d91f7945f58403f8f6898f8c7f"
format_stamp: "Formatted at 2026-09-29 18:37:02 on dist-test-slave-qcpv"
server_key: "eaae44cf3092b3fbba743593d32c60c1"
server_key_iv: "ce8a2358d14a5e3161741a9bbe9ef97a"
server_key_version: "encryptionkey@0"
I20260929 18:37:02.684244 19302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:02.708781 19302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:02.713155 19302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:02.713428 19302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:02.724125 19302 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:43313
I20260929 18:37:02.724493 19369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:43313 every 8 connection(s)
I20260929 18:37:02.725644 19302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:37:02.727906 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 19302
I20260929 18:37:02.728053 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:37:02.728415 29501 external_mini_cluster.cc:1428] Setting key c0846ee51ab899d1905e1fb9f9064aeb
I20260929 18:37:02.729159 19370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:02.734958 19370 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43313 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.207.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.125@KRBTEST.COM" created.
Sep 29 18:37:02 dist-test-slave-qcpv krb5kdc[19289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707022, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:37:02.756695 19374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:02.739829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49670 (local address 127.28.207.126:43313)
0929 18:37:02.740458 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:02.740486 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:02.740640 (+ 154us) server_negotiation.cc:407] Connection header received
0929 18:37:02.740820 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:02.740824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:02.741075 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:02.741293 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:02.742514 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:02.743626 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:02.745076 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:02.745888 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:02.746134 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:02.749301 (+ 3167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:02.749313 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:02.749337 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:02.749385 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:02.751962 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:02.752801 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:02.752807 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:02.752808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:02.752947 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:02.753436 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:02.753439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:02.753440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:02.754074 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:02.754215 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:02.755093 (+ 878us) server_negotiation.cc:299] Negotiation successful
0929 18:37:02.755399 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":172,"threads_started":1}
I20260929 18:37:02.756707 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:02.738151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43313 (local address 127.0.0.1:49670)
0929 18:37:02.739169 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:02.739264 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:02.740206 (+ 942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:02.741398 (+ 1192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:02.741405 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:02.741823 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:02.742300 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:02.742318 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:02.743824 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:02.743831 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:02.744865 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:02.744880 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:02.746149 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:02.746153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:02.746296 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:02.747158 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:02.747178 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:02.749057 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:02.752212 (+ 3155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:02.752225 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:02.752242 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:02.752621 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:02.753088 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:02.753091 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:02.753092 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:02.753275 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:02.754253 (+ 978us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:02.754259 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:02.754550 (+ 291us) client_negotiation.cc:769] Sending connection context
0929 18:37:02.754969 (+ 419us) client_negotiation.cc:240] Negotiation successful
0929 18:37:02.755586 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"mutex_wait_us":207,"thread_start_us":316,"threads_started":1}
I20260929 18:37:02.760115 19370 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.207.125" port: 43445 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:37:02.761209 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:02.760827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.125:43445 (local address 127.0.0.1:54802)
0929 18:37:02.760998 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:02.761098 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.125:43445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260929 18:37:02.761543 19318 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.207.125:43445: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.125.keytab.
Entry for principal kudu/127.28.207.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.125.keytab.
I20260929 18:37:02.763125 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:02.762794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.125:43445 (local address 127.0.0.1:54810)
0929 18:37:02.762977 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:02.763048 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.125:43445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260929 18:37:02.763388 19370 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.207.125:43445: Network error: Client connection negotiation failed: client connection to 127.28.207.125:43445: connect: Connection refused (error 111)
I20260929 18:37:02.763545 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.125:43445
--webserver_interface=127.28.207.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.125.keytab
--principal=kudu/127.28.207.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:45399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:43313,127.28.207.125:43445,127.28.207.124:39019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:37:02.812741 19370 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.207.125" port: 43445 } attempt: 1
I20260929 18:37:02.813696 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:02.813401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.125:43445 (local address 127.0.0.1:54820)
0929 18:37:02.813549 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:02.813616 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.125:43445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 18:37:02.814466 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:02.814242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.125:43445 (local address 127.0.0.1:54822)
0929 18:37:02.814358 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:02.814406 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.125:43445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 18:37:02.814634 19370 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.207.125:43445: Network error: Client connection negotiation failed: client connection to 127.28.207.125:43445: connect: Connection refused (error 111)
I20260929 18:37:02.882905 19370 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.207.125" port: 43445 } attempt: 2
I20260929 18:37:02.883810 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:02.883483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.125:43445 (local address 127.0.0.1:54830)
0929 18:37:02.883668 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:02.883739 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.125:43445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 18:37:02.884873 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:02.884612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.125:43445 (local address 127.0.0.1:54840)
0929 18:37:02.884766 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:02.884820 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.125:43445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260929 18:37:02.885193 19370 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.207.125:43445: Network error: Client connection negotiation failed: client connection to 127.28.207.125:43445: connect: Connection refused (error 111)
W20260929 18:37:02.912503 19377 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 18:37:02.912935 19377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:02.912999 19377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:02.913085 19377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:02.918179 19377 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:37:02.918306 19377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:02.918329 19377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:02.918347 19377 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:37:02.918365 19377 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:37:02.925103 19377 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:45399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:43313,127.28.207.125:43445,127.28.207.124:39019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.125.keytab
--principal=kudu/127.28.207.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.125:43445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.207.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:02.926752 19377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:02.928745 19377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:02.939505 19385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:02.939682 19382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:02.939862 19377 server_base.cc:1034] running on GCE node
W20260929 18:37:02.940003 19383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:02.940735 19377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:02.942111 19377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:02.945458 19377 hybrid_clock.cc:648] HybridClock initialized: now 1790707022945367 us; error 132 us; skew 500 ppm
Sep 29 18:37:02 dist-test-slave-qcpv krb5kdc[19289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707022, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:02.949206 19377 init.cc:377] Logged in from keytab as kudu/127.28.207.125@KRBTEST.COM (short username kudu)
I20260929 18:37:02.950708 19377 webserver.cc:460] Webserver started at http://127.28.207.125:43305/ using document root <none> and password file <none>
I20260929 18:37:02.951534 19377 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:02.951606 19377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:02.951794 19377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:02.954054 19377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-1/data/instance:
uuid: "baabbcb3cd2345fe90ba6e60a651c5a0"
format_stamp: "Formatted at 2026-09-29 18:37:02 on dist-test-slave-qcpv"
server_key: "4059451608f6b62a4586a22afd9f4511"
server_key_iv: "a9fde0a2c7e67a8bf0bfc260a31f1027"
server_key_version: "encryptionkey@0"
I20260929 18:37:02.954599 19377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-1/wal/instance:
uuid: "baabbcb3cd2345fe90ba6e60a651c5a0"
format_stamp: "Formatted at 2026-09-29 18:37:02 on dist-test-slave-qcpv"
server_key: "4059451608f6b62a4586a22afd9f4511"
server_key_iv: "a9fde0a2c7e67a8bf0bfc260a31f1027"
server_key_version: "encryptionkey@0"
I20260929 18:37:02.959411 19377 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 18:37:02.962432 19392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:02.964394 19377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:37:02.964565 19377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-1/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-1/wal
uuid: "baabbcb3cd2345fe90ba6e60a651c5a0"
format_stamp: "Formatted at 2026-09-29 18:37:02 on dist-test-slave-qcpv"
server_key: "4059451608f6b62a4586a22afd9f4511"
server_key_iv: "a9fde0a2c7e67a8bf0bfc260a31f1027"
server_key_version: "encryptionkey@0"
I20260929 18:37:02.964848 19377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:02.976404 19370 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.207.125" port: 43445 } attempt: 3
I20260929 18:37:02.978035 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:02.977656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.125:43445 (local address 127.0.0.1:54842)
0929 18:37:02.977836 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:02.977911 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.125:43445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 18:37:02.979102 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:02.978833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.125:43445 (local address 127.0.0.1:54858)
0929 18:37:02.978973 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:02.979025 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.125:43445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 18:37:02.979300 19370 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.207.125:43445: Network error: Client connection negotiation failed: client connection to 127.28.207.125:43445: connect: Connection refused (error 111)
I20260929 18:37:02.990273 19377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:02.994298 19377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:02.994570 19377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:03.005712 19377 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.125:43445
I20260929 18:37:03.006111 19444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.125:43445 every 8 connection(s)
I20260929 18:37:03.007651 19377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-1/data/info.pb
I20260929 18:37:03.012748 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 19377
I20260929 18:37:03.012904 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-1/wal/instance
I20260929 18:37:03.013311 29501 external_mini_cluster.cc:1428] Setting key 6a736f3c22dc9c006fac8800d7b56f3b
I20260929 18:37:03.013820 19445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:03.022743 19445 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43313 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.207.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.124@KRBTEST.COM" created.
Sep 29 18:37:03 dist-test-slave-qcpv krb5kdc[19289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707022, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.125@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:37:03.045581 19374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:03.026079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49684 (local address 127.28.207.126:43313)
0929 18:37:03.028145 (+ 2066us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:03.028152 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:03.028419 (+ 267us) server_negotiation.cc:407] Connection header received
0929 18:37:03.029772 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:03.029777 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:03.029842 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:03.030038 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:03.032534 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.033213 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.034813 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.035307 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.035489 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:03.039434 (+ 3945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:03.039441 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:03.039445 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:03.039492 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:03.042197 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:03.043383 (+ 1186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:03.043387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:03.043388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:03.043491 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:03.044032 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:03.044035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:03.044036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:03.044317 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:03.044464 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:03.045252 (+ 788us) server_negotiation.cc:299] Negotiation successful
0929 18:37:03.045439 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1882}
I20260929 18:37:03.046970 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:03.026699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43313 (local address 127.0.0.1:49684)
0929 18:37:03.028081 (+ 1382us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:03.028171 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:03.029449 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:03.030491 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:03.030501 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:03.031143 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:03.032218 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:03.032247 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.033498 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.033505 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:03.034638 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:03.034652 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.035581 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.035585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:03.035788 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:03.036807 (+ 1019us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:03.036829 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:03.039138 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:03.042455 (+ 3317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:03.042464 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:03.042486 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:03.043205 (+ 719us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:03.043665 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:03.043676 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:03.043678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:03.043892 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:03.044491 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:03.044500 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:03.044897 (+ 397us) client_negotiation.cc:769] Sending connection context
0929 18:37:03.045133 (+ 236us) client_negotiation.cc:240] Negotiation successful
0929 18:37:03.045617 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":212,"threads_started":1}
I20260929 18:37:03.049104 19445 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.207.125" port: 43445 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.124.keytab.
Entry for principal kudu/127.28.207.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.124.keytab.
I20260929 18:37:03.055725 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.124:39019
--webserver_interface=127.28.207.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.124.keytab
--principal=kudu/127.28.207.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:45399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:43313,127.28.207.125:43445,127.28.207.124:39019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 18:37:03 dist-test-slave-qcpv krb5kdc[19289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707022, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.125@KRBTEST.COM for kudu/127.28.207.125@KRBTEST.COM
I20260929 18:37:03.062450 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:03.049945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.125:43445 (local address 127.0.0.1:54874)
0929 18:37:03.050106 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:03.050138 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:03.050254 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:03.051126 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:03.051130 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:03.051373 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:03.051512 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:03.051519 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.052761 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.052765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:03.053435 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:03.053445 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.054232 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.054235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:03.054336 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:03.055347 (+ 1011us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:03.055363 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:03.057197 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:03.059738 (+ 2541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:03.059745 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:03.059748 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:03.059988 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:03.060440 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:03.060443 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:03.060445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:03.060543 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:03.061235 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:03.061238 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:03.061332 (+ 94us) client_negotiation.cc:769] Sending connection context
0929 18:37:03.061579 (+ 247us) client_negotiation.cc:240] Negotiation successful
0929 18:37:03.061766 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 18:37:03.062493 19450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:03.050079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54874 (local address 127.28.207.125:43445)
0929 18:37:03.050579 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:03.050589 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:03.050610 (+ 21us) server_negotiation.cc:407] Connection header received
0929 18:37:03.050646 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:03.050652 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:03.050933 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:03.051084 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:03.051693 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.052606 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.053587 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.054060 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.054170 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:03.057409 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:03.057428 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:03.057448 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:03.057470 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:03.059594 (+ 2124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:03.060219 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:03.060232 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:03.060233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:03.060327 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:03.060638 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:03.060647 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:03.060649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:03.061095 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:03.061289 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:03.061683 (+ 394us) server_negotiation.cc:299] Negotiation successful
0929 18:37:03.061842 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":241,"threads_started":1}
I20260929 18:37:03.065862 19445 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.207.124" port: 39019 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:37:03.067058 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:03.066539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:39019 (local address 127.0.0.1:33652)
0929 18:37:03.066758 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:03.066942 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.124:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260929 18:37:03.067404 19395 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.207.124:39019: connect: Connection refused (error 111)
I20260929 18:37:03.068621 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:03.068306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:39019 (local address 127.0.0.1:33664)
0929 18:37:03.068474 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:03.068537 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.124:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260929 18:37:03.068861 19445 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.207.124:39019: Network error: Client connection negotiation failed: client connection to 127.28.207.124:39019: connect: Connection refused (error 111)
I20260929 18:37:03.118314 19445 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.207.124" port: 39019 } attempt: 1
I20260929 18:37:03.119599 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:03.119213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:39019 (local address 127.0.0.1:33668)
0929 18:37:03.119412 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:03.119497 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.124:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 18:37:03.120483 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:03.120239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:39019 (local address 127.0.0.1:33670)
0929 18:37:03.120357 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:03.120416 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.124:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 18:37:03.120692 19445 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.207.124:39019: Network error: Client connection negotiation failed: client connection to 127.28.207.124:39019: connect: Connection refused (error 111)
I20260929 18:37:03.122735 19370 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.207.125" port: 43445 } attempt: 4
Sep 29 18:37:03 dist-test-slave-qcpv krb5kdc[19289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707022, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for kudu/127.28.207.125@KRBTEST.COM
I20260929 18:37:03.135963 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:03.124169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.125:43445 (local address 127.0.0.1:54890)
0929 18:37:03.124521 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:03.124568 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:03.124773 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:03.125679 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:03.125682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:03.125913 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:03.126056 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:03.126063 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.127364 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.127368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:03.128012 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:03.128021 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.128887 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.128892 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:03.129025 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:03.129600 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:03.129619 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:03.131326 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:03.134128 (+ 2802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:03.134132 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:03.134135 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:03.134382 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:03.134717 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:03.134722 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:03.134724 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:03.134827 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:03.135328 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:03.135331 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:03.135449 (+ 118us) client_negotiation.cc:769] Sending connection context
0929 18:37:03.135564 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 18:37:03.135751 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230}
I20260929 18:37:03.135963 19450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:03.124313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54890 (local address 127.28.207.125:43445)
0929 18:37:03.124514 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:03.124520 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:03.124715 (+ 195us) server_negotiation.cc:407] Connection header received
0929 18:37:03.125278 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:03.125283 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:03.125386 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:03.125530 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:03.126193 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.127207 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.128376 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.128729 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.128867 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:03.131929 (+ 3062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:03.131933 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:03.131935 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:03.131971 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:03.133976 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:03.134498 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:03.134502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:03.134504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:03.134590 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:03.134977 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:03.134980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:03.134982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:03.135216 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:03.135331 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:03.135617 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 18:37:03.135751 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 18:37:03.137226 19370 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.207.124" port: 39019 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:37:03.138144 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:03.137847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:39019 (local address 127.0.0.1:33686)
0929 18:37:03.138000 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:03.138060 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.124:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 18:37:03.139732 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:03.139296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:39019 (local address 127.0.0.1:33694)
0929 18:37:03.139547 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:03.139636 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.124:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
W20260929 18:37:03.139900 19370 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.207.124:39019: Network error: Client connection negotiation failed: client connection to 127.28.207.124:39019: connect: Connection refused (error 111)
W20260929 18:37:03.185999 19451 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 18:37:03.186367 19451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:03.186393 19451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:03.186479 19451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 18:37:03.188923 19445 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.207.124" port: 39019 } attempt: 2
I20260929 18:37:03.190089 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:03.189697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:39019 (local address 127.0.0.1:33702)
0929 18:37:03.189883 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:03.189968 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.124:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 18:37:03.190837 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:03.190616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:39019 (local address 127.0.0.1:33708)
0929 18:37:03.190722 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:03.190773 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.124:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260929 18:37:03.191035 19445 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.207.124:39019: Network error: Client connection negotiation failed: client connection to 127.28.207.124:39019: connect: Connection refused (error 111)
W20260929 18:37:03.191313 19451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:37:03.191414 19451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:03.191457 19451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:03.191517 19451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:37:03.191556 19451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:37:03.197494 19451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:45399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:43313,127.28.207.125:43445,127.28.207.124:39019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.124.keytab
--principal=kudu/127.28.207.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.124:39019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.207.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:03.198761 19451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:03.199113 19370 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.207.124" port: 39019 } attempt: 1
I20260929 18:37:03.199947 19451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 18:37:03.200340 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:03.199983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:39019 (local address 127.0.0.1:33710)
0929 18:37:03.200162 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:03.200242 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.124:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 18:37:03.201136 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:03.200907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:39019 (local address 127.0.0.1:33720)
0929 18:37:03.201024 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:03.201080 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.124:39019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 18:37:03.201287 19370 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.207.124:39019: Network error: Client connection negotiation failed: client connection to 127.28.207.124:39019: connect: Connection refused (error 111)
W20260929 18:37:03.210584 19458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:03.210584 19457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:03.210585 19460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:03.210701 19451 server_base.cc:1034] running on GCE node
I20260929 18:37:03.211725 19451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:03.212971 19451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:03.214267 19451 hybrid_clock.cc:648] HybridClock initialized: now 1790707023214193 us; error 85 us; skew 500 ppm
Sep 29 18:37:03 dist-test-slave-qcpv krb5kdc[19289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707023, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:03.217774 19451 init.cc:377] Logged in from keytab as kudu/127.28.207.124@KRBTEST.COM (short username kudu)
I20260929 18:37:03.218958 19451 webserver.cc:460] Webserver started at http://127.28.207.124:33141/ using document root <none> and password file <none>
I20260929 18:37:03.219595 19451 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:03.219708 19451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:03.219959 19451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:03.222220 19451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-2/data/instance:
uuid: "307d9d3a9bfd400c9a4b663f53ac824e"
format_stamp: "Formatted at 2026-09-29 18:37:03 on dist-test-slave-qcpv"
server_key: "8fc99c55cdc7df422dc3853067ded3df"
server_key_iv: "6d7500c310a783ebd9adf59d5cd75612"
server_key_version: "encryptionkey@0"
I20260929 18:37:03.222950 19451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-2/wal/instance:
uuid: "307d9d3a9bfd400c9a4b663f53ac824e"
format_stamp: "Formatted at 2026-09-29 18:37:03 on dist-test-slave-qcpv"
server_key: "8fc99c55cdc7df422dc3853067ded3df"
server_key_iv: "6d7500c310a783ebd9adf59d5cd75612"
server_key_version: "encryptionkey@0"
I20260929 18:37:03.227522 19451 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 18:37:03.230551 19467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:03.232419 19451 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 18:37:03.232640 19451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-2/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-2/wal
uuid: "307d9d3a9bfd400c9a4b663f53ac824e"
format_stamp: "Formatted at 2026-09-29 18:37:03 on dist-test-slave-qcpv"
server_key: "8fc99c55cdc7df422dc3853067ded3df"
server_key_iv: "6d7500c310a783ebd9adf59d5cd75612"
server_key_version: "encryptionkey@0"
I20260929 18:37:03.232791 19451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:03.247936 19451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:03.251830 19451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:03.252071 19451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:03.261511 19451 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.124:39019
I20260929 18:37:03.262027 19519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.124:39019 every 8 connection(s)
I20260929 18:37:03.262800 19451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-2/data/info.pb
I20260929 18:37:03.264035 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 19451
I20260929 18:37:03.264231 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790706718260899-29501-0/minicluster-data/master-2/wal/instance
I20260929 18:37:03.264554 29501 external_mini_cluster.cc:1428] Setting key a5e3b67fe7edf56807e9af1a4df4f9f5
I20260929 18:37:03.267303 19520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:03.268484 19370 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.207.124" port: 39019 } attempt: 2
I20260929 18:37:03.273613 19520 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43313 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:37:03 dist-test-slave-qcpv krb5kdc[19289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707022, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for kudu/127.28.207.124@KRBTEST.COM
Sep 29 18:37:03 dist-test-slave-qcpv krb5kdc[19289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.207.124@KRBTEST.COM
I20260929 18:37:03.282219 19445 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.207.124" port: 39019 } attempt: 3
I20260929 18:37:03.283811 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:03.269162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:39019 (local address 127.0.0.1:33748)
0929 18:37:03.269428 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:03.269447 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:03.269556 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:03.270685 (+ 1129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:03.270688 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:03.270971 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:03.271106 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:03.271113 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.272498 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.272503 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:03.273343 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:03.273358 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.274699 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.274704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:03.274838 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:03.275499 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:03.275515 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:03.278038 (+ 2523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:03.281277 (+ 3239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:03.281282 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:03.281284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:03.281564 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:03.282048 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:03.282051 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:03.282054 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:03.282199 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:03.283200 (+ 1001us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:03.283204 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:03.283336 (+ 132us) client_negotiation.cc:769] Sending connection context
0929 18:37:03.283436 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 18:37:03.283609 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168}
Sep 29 18:37:03 dist-test-slave-qcpv krb5kdc[19289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707023, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.124@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:37:03.285879 19523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:03.266241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33736 (local address 127.28.207.124:39019)
0929 18:37:03.267008 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:03.267029 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:03.267085 (+ 56us) server_negotiation.cc:407] Connection header received
0929 18:37:03.267950 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:03.267973 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:03.268417 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:03.268838 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:03.270260 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.272081 (+ 1821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.273337 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.274511 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.275152 (+ 641us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:03.278337 (+ 3185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:03.278342 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:03.278345 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:03.278374 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:03.281012 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:03.281694 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:03.281702 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:03.281711 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:03.281924 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:03.282311 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:03.282314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:03.282316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:03.283032 (+ 716us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:03.284221 (+ 1189us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:03.284301 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 18:37:03.284634 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":187,"server-negotiator.queue_time_us":491,"spinlock_wait_cycles":2304,"thread_start_us":225,"threads_started":1}
I20260929 18:37:03.285879 19524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:03.269941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33748 (local address 127.28.207.124:39019)
0929 18:37:03.270410 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:03.270417 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:03.270431 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:37:03.270471 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:03.270473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:03.270508 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:03.270635 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:03.271238 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.272075 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.273486 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.274511 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.274717 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:03.278273 (+ 3556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:03.278283 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:03.278296 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:03.278335 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:03.281150 (+ 2815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:03.281694 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:03.281702 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:03.281711 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:03.281924 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:03.282332 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:03.282337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:03.282339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:03.283044 (+ 705us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:03.284065 (+ 1021us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:03.284263 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 18:37:03.284766 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":250,"server-negotiator.queue_time_us":397,"spinlock_wait_cycles":21376,"thread_start_us":232,"threads_started":1}
I20260929 18:37:03.289501 19374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:03.277067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49692 (local address 127.28.207.126:43313)
0929 18:37:03.277188 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:03.277191 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:03.277203 (+ 12us) server_negotiation.cc:407] Connection header received
0929 18:37:03.277240 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:03.277242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:03.277278 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:03.277365 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:03.278518 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.279267 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.280667 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.281045 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.281172 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:03.284942 (+ 3770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:03.284946 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:03.284949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:03.284983 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:03.287182 (+ 2199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:03.287799 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:03.287803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:03.287805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:03.287926 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:03.288283 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:03.288287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:03.288289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:03.288668 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:03.288742 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:03.289227 (+ 485us) server_negotiation.cc:299] Negotiation successful
0929 18:37:03.289364 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 18:37:03.290207 19525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:03.276084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:43313 (local address 127.0.0.1:49692)
0929 18:37:03.276701 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:03.276781 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:03.276948 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:03.277488 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:03.277496 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:03.278118 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:03.278370 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:03.278379 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.279407 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.279414 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:03.280508 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:03.280521 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.281316 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.281320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:03.281439 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:03.282014 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:03.282033 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:03.284766 (+ 2733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:03.287325 (+ 2559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:03.287337 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:03.287344 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:03.287677 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:03.288049 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:03.288053 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:03.288054 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:03.288159 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:03.288821 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:03.288828 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:03.288960 (+ 132us) client_negotiation.cc:769] Sending connection context
0929 18:37:03.289836 (+ 876us) client_negotiation.cc:240] Negotiation successful
0929 18:37:03.290069 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":143,"threads_started":1}
Sep 29 18:37:03 dist-test-slave-qcpv krb5kdc[19289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707022, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.125@KRBTEST.COM for kudu/127.28.207.124@KRBTEST.COM
I20260929 18:37:03.291435 19370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b1ad8d91f7945f58403f8f6898f8c7f: Bootstrap starting.
I20260929 18:37:03.291973 19520 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.207.125" port: 43445 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:37:03.294330 19370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b1ad8d91f7945f58403f8f6898f8c7f: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:03.296216 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:03.282899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:39019 (local address 127.0.0.1:33756)
0929 18:37:03.283923 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:03.283941 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:03.284030 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:03.285453 (+ 1423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:03.285456 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:03.285657 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:03.285764 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:03.285770 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.286951 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.286956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:03.287846 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:03.287862 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.289194 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.289203 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:03.289330 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:03.289984 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:03.290000 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:03.291678 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:03.294456 (+ 2778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:03.294460 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:03.294463 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:03.294742 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:03.295134 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:03.295138 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:03.295140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:03.295233 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:03.295731 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:03.295735 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:03.295833 (+ 98us) client_negotiation.cc:769] Sending connection context
0929 18:37:03.295912 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 18:37:03.296061 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467}
I20260929 18:37:03.296802 19370 log.cc:824] T 00000000000000000000000000000000 P 3b1ad8d91f7945f58403f8f6898f8c7f: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:03.297349 19526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:03.283118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33756 (local address 127.28.207.124:39019)
0929 18:37:03.285169 (+ 2051us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:03.285180 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:03.285195 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:37:03.285234 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:03.285240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:03.285311 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:03.285427 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:03.285891 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.286763 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.288634 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.289009 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.289134 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:03.291845 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:03.291848 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:03.291851 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:03.291876 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:03.294328 (+ 2452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:03.294916 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:03.294922 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:03.294926 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:03.295021 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:03.295366 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:03.295369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:03.295370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:03.295615 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:03.296570 (+ 955us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:03.296649 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 18:37:03.297090 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1961,"thread_start_us":132,"threads_started":1}
Sep 29 18:37:03 dist-test-slave-qcpv krb5kdc[19289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707023, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.124@KRBTEST.COM for kudu/127.28.207.125@KRBTEST.COM
I20260929 18:37:03.300724 19370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b1ad8d91f7945f58403f8f6898f8c7f: No bootstrap required, opened a new log
Sep 29 18:37:03 dist-test-slave-qcpv krb5kdc[19289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:37:03.302522 19450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:03.292659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54896 (local address 127.28.207.125:43445)
0929 18:37:03.292908 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:03.292913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:03.293010 (+ 97us) server_negotiation.cc:407] Connection header received
0929 18:37:03.293159 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:03.293162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:03.293210 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:03.293458 (+ 248us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:03.293856 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.294663 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.295864 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.296247 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.296347 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:03.298690 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:03.298694 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:03.298697 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:03.298730 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:03.300466 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:03.301044 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:03.301048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:03.301049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:03.301142 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:03.301450 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:03.301453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:03.301455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:03.301772 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:03.301848 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:03.302198 (+ 350us) server_negotiation.cc:299] Negotiation successful
0929 18:37:03.302341 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 18:37:03.303241 19445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baabbcb3cd2345fe90ba6e60a651c5a0: Bootstrap starting.
I20260929 18:37:03.303421 19525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:03.292581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.125:43445 (local address 127.0.0.1:54896)
0929 18:37:03.292924 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:03.292940 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:03.293054 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:03.293364 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:03.293369 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:03.293545 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:03.293696 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:03.293704 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.294791 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.294795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:03.295696 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:03.295705 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.296537 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.296540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:03.296635 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:03.297098 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:03.297113 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:03.298525 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:03.300603 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:03.300607 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:03.300609 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:03.300927 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:03.301261 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:03.301264 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:03.301266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:03.301349 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:03.301923 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:03.301926 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:03.302023 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 18:37:03.303077 (+ 1054us) client_negotiation.cc:240] Negotiation successful
0929 18:37:03.303257 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20260929 18:37:03.304767 19520 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.207.124" port: 39019 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:37:03.305655 19370 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b1ad8d91f7945f58403f8f6898f8c7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1ad8d91f7945f58403f8f6898f8c7f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43313 } } peers { permanent_uuid: "baabbcb3cd2345fe90ba6e60a651c5a0" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 43445 } } peers { permanent_uuid: "307d9d3a9bfd400c9a4b663f53ac824e" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 39019 } }
I20260929 18:37:03.305953 19370 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b1ad8d91f7945f58403f8f6898f8c7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b1ad8d91f7945f58403f8f6898f8c7f, State: Initialized, Role: FOLLOWER
I20260929 18:37:03.306423 19370 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b1ad8d91f7945f58403f8f6898f8c7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1ad8d91f7945f58403f8f6898f8c7f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43313 } } peers { permanent_uuid: "baabbcb3cd2345fe90ba6e60a651c5a0" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 43445 } } peers { permanent_uuid: "307d9d3a9bfd400c9a4b663f53ac824e" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 39019 } }
I20260929 18:37:03.307050 19528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b1ad8d91f7945f58403f8f6898f8c7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1ad8d91f7945f58403f8f6898f8c7f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43313 } } peers { permanent_uuid: "baabbcb3cd2345fe90ba6e60a651c5a0" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 43445 } } peers { permanent_uuid: "307d9d3a9bfd400c9a4b663f53ac824e" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 39019 } } }
I20260929 18:37:03.307272 19528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b1ad8d91f7945f58403f8f6898f8c7f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:37:03.307374 19445 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P baabbcb3cd2345fe90ba6e60a651c5a0: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:03.307883 19374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:03.295248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49702 (local address 127.28.207.126:43313)
0929 18:37:03.295369 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:03.295374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:03.295573 (+ 199us) server_negotiation.cc:407] Connection header received
0929 18:37:03.295714 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:03.295717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:03.295773 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:03.295860 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:03.296849 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.297635 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.298952 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.299353 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.299490 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:03.302923 (+ 3433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:03.302927 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:03.302930 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:03.302965 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:03.304717 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:03.305849 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:03.305853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:03.305856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:03.305949 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:03.306243 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:03.306246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:03.306248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:03.306497 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:03.306701 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:03.307503 (+ 802us) server_negotiation.cc:299] Negotiation successful
0929 18:37:03.307657 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 18:37:03.308065 19370 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b1ad8d91f7945f58403f8f6898f8c7f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:37:03.308563 19445 log.cc:824] T 00000000000000000000000000000000 P baabbcb3cd2345fe90ba6e60a651c5a0: Log is configured to *not* fsync() on all Append() calls
W20260929 18:37:03.313807 19540 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3b1ad8d91f7945f58403f8f6898f8c7f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:37:03.314018 19540 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:37:03.316146 19445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baabbcb3cd2345fe90ba6e60a651c5a0: No bootstrap required, opened a new log
Sep 29 18:37:03 dist-test-slave-qcpv krb5kdc[19289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.207.125@KRBTEST.COM
Sep 29 18:37:03 dist-test-slave-qcpv krb5kdc[19289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707023, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.124@KRBTEST.COM for kudu/127.28.207.124@KRBTEST.COM
I20260929 18:37:03.313791 19302 master_runner.cc:186] Error getting master registration for 127.28.207.125:43445: OK, instance_id { permanent_uuid: "baabbcb3cd2345fe90ba6e60a651c5a0" instance_seqno: 1790707023002321 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 18:37:03.318673 19302 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:37:03.321880 19525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:03.305337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:39019 (local address 127.0.0.1:33772)
0929 18:37:03.305489 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:03.305506 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:03.305597 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:03.309476 (+ 3879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:03.309480 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:03.309695 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:03.309985 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:03.309999 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.313637 (+ 3638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.313642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:03.314496 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:03.314506 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.315188 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.315195 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:03.315376 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:03.315941 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:03.315956 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:03.317850 (+ 1894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:03.320432 (+ 2582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:03.320435 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:03.320437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:03.320627 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:03.320905 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:03.320908 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:03.320909 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:03.320986 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:03.321424 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:03.321426 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:03.321514 (+ 88us) client_negotiation.cc:769] Sending connection context
0929 18:37:03.321607 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 18:37:03.321749 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 18:37:03.322295 19450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:03.309424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54900 (local address 127.28.207.125:43445)
0929 18:37:03.309554 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:03.309559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:03.309928 (+ 369us) server_negotiation.cc:407] Connection header received
0929 18:37:03.310174 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:03.310177 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:03.310230 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:03.310288 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:03.311267 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.312016 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.313233 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.313542 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.313620 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:03.317068 (+ 3448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:03.317073 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:03.317075 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:03.317107 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:03.319044 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:03.319509 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:03.319514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:03.319516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:03.319605 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:03.319999 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:03.320004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:03.320006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:03.320708 (+ 702us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:03.321398 (+ 690us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:03.321500 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 18:37:03.321904 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 18:37:03.322454 19526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:03.305569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33772 (local address 127.28.207.124:39019)
0929 18:37:03.309189 (+ 3620us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:03.309196 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:03.309211 (+ 15us) server_negotiation.cc:407] Connection header received
0929 18:37:03.309261 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:03.309267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:03.309344 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:03.311533 (+ 2189us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:03.311640 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.312392 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.314637 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.315008 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.315127 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:03.318050 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:03.318055 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:03.318060 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:03.318096 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:03.320308 (+ 2212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:03.320727 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:03.320732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:03.320734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:03.320817 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:03.321101 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:03.321103 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:03.321105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:03.321331 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:03.322079 (+ 748us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:03.322151 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 18:37:03.322268 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3516}
I20260929 18:37:03.323107 19445 raft_consensus.cc:348] T 00000000000000000000000000000000 P baabbcb3cd2345fe90ba6e60a651c5a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1ad8d91f7945f58403f8f6898f8c7f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43313 } } peers { permanent_uuid: "baabbcb3cd2345fe90ba6e60a651c5a0" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 43445 } } peers { permanent_uuid: "307d9d3a9bfd400c9a4b663f53ac824e" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 39019 } }
I20260929 18:37:03.323455 19445 raft_consensus.cc:729] T 00000000000000000000000000000000 P baabbcb3cd2345fe90ba6e60a651c5a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baabbcb3cd2345fe90ba6e60a651c5a0, State: Initialized, Role: FOLLOWER
I20260929 18:37:03.323916 19445 consensus_queue.cc:260] T 00000000000000000000000000000000 P baabbcb3cd2345fe90ba6e60a651c5a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1ad8d91f7945f58403f8f6898f8c7f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43313 } } peers { permanent_uuid: "baabbcb3cd2345fe90ba6e60a651c5a0" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 43445 } } peers { permanent_uuid: "307d9d3a9bfd400c9a4b663f53ac824e" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 39019 } }
I20260929 18:37:03.324749 19542 sys_catalog.cc:455] T 00000000000000000000000000000000 P baabbcb3cd2345fe90ba6e60a651c5a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1ad8d91f7945f58403f8f6898f8c7f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43313 } } peers { permanent_uuid: "baabbcb3cd2345fe90ba6e60a651c5a0" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 43445 } } peers { permanent_uuid: "307d9d3a9bfd400c9a4b663f53ac824e" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 39019 } } }
I20260929 18:37:03.324968 19542 sys_catalog.cc:458] T 00000000000000000000000000000000 P baabbcb3cd2345fe90ba6e60a651c5a0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:37:03.326349 19445 sys_catalog.cc:564] T 00000000000000000000000000000000 P baabbcb3cd2345fe90ba6e60a651c5a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:37:03.326704 19520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 307d9d3a9bfd400c9a4b663f53ac824e: Bootstrap starting.
I20260929 18:37:03.327198 29501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:37:03.329141 19344 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3b1ad8d91f7945f58403f8f6898f8c7f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49702
I20260929 18:37:03.329460 19344 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b1ad8d91f7945f58403f8f6898f8c7f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 18:37:03.329525 19520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 307d9d3a9bfd400c9a4b663f53ac824e: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:03.329578 19344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b1ad8d91f7945f58403f8f6898f8c7f [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:03.330435 19520 log.cc:824] T 00000000000000000000000000000000 P 307d9d3a9bfd400c9a4b663f53ac824e: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:03.331180 19344 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b1ad8d91f7945f58403f8f6898f8c7f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1ad8d91f7945f58403f8f6898f8c7f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43313 } } peers { permanent_uuid: "baabbcb3cd2345fe90ba6e60a651c5a0" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 43445 } } peers { permanent_uuid: "307d9d3a9bfd400c9a4b663f53ac824e" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 39019 } }
I20260929 18:37:03.332368 19344 leader_election.cc:288] T 00000000000000000000000000000000 P 3b1ad8d91f7945f58403f8f6898f8c7f [CANDIDATE]: Term 1 election: Requested vote from peers baabbcb3cd2345fe90ba6e60a651c5a0 (127.28.207.125:43445), 307d9d3a9bfd400c9a4b663f53ac824e (127.28.207.124:39019)
I20260929 18:37:03.333256 19520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 307d9d3a9bfd400c9a4b663f53ac824e: No bootstrap required, opened a new log
I20260929 18:37:03.333153 19419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3b1ad8d91f7945f58403f8f6898f8c7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "baabbcb3cd2345fe90ba6e60a651c5a0"
I20260929 18:37:03.333406 19419 raft_consensus.cc:3037] T 00000000000000000000000000000000 P baabbcb3cd2345fe90ba6e60a651c5a0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:03.334841 19419 raft_consensus.cc:2454] T 00000000000000000000000000000000 P baabbcb3cd2345fe90ba6e60a651c5a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b1ad8d91f7945f58403f8f6898f8c7f in term 1.
I20260929 18:37:03.335847 19318 leader_election.cc:302] T 00000000000000000000000000000000 P 3b1ad8d91f7945f58403f8f6898f8c7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b1ad8d91f7945f58403f8f6898f8c7f, baabbcb3cd2345fe90ba6e60a651c5a0; no voters:
I20260929 18:37:03.336328 19528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b1ad8d91f7945f58403f8f6898f8c7f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:37:03.336745 19528 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b1ad8d91f7945f58403f8f6898f8c7f [term 1 LEADER]: Becoming Leader. State: Replica: 3b1ad8d91f7945f58403f8f6898f8c7f, State: Running, Role: LEADER
I20260929 18:37:03.337277 19528 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b1ad8d91f7945f58403f8f6898f8c7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1ad8d91f7945f58403f8f6898f8c7f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43313 } } peers { permanent_uuid: "baabbcb3cd2345fe90ba6e60a651c5a0" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 43445 } } peers { permanent_uuid: "307d9d3a9bfd400c9a4b663f53ac824e" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 39019 } }
I20260929 18:37:03.338630 19520 raft_consensus.cc:348] T 00000000000000000000000000000000 P 307d9d3a9bfd400c9a4b663f53ac824e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1ad8d91f7945f58403f8f6898f8c7f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43313 } } peers { permanent_uuid: "baabbcb3cd2345fe90ba6e60a651c5a0" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 43445 } } peers { permanent_uuid: "307d9d3a9bfd400c9a4b663f53ac824e" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 39019 } }
I20260929 18:37:03.338997 19520 raft_consensus.cc:729] T 00000000000000000000000000000000 P 307d9d3a9bfd400c9a4b663f53ac824e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 307d9d3a9bfd400c9a4b663f53ac824e, State: Initialized, Role: FOLLOWER
I20260929 18:37:03.339373 19520 consensus_queue.cc:260] T 00000000000000000000000000000000 P 307d9d3a9bfd400c9a4b663f53ac824e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1ad8d91f7945f58403f8f6898f8c7f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43313 } } peers { permanent_uuid: "baabbcb3cd2345fe90ba6e60a651c5a0" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 43445 } } peers { permanent_uuid: "307d9d3a9bfd400c9a4b663f53ac824e" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 39019 } }
I20260929 18:37:03.340930 19520 sys_catalog.cc:564] T 00000000000000000000000000000000 P 307d9d3a9bfd400c9a4b663f53ac824e [sys.catalog]: configured and running, proceeding with master startup.
W20260929 18:37:03.341259 19559 catalog_manager.cc:1558] T 00000000000000000000000000000000 P baabbcb3cd2345fe90ba6e60a651c5a0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:37:03.341374 19559 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:37:03.341655 19561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 307d9d3a9bfd400c9a4b663f53ac824e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1ad8d91f7945f58403f8f6898f8c7f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43313 } } peers { permanent_uuid: "baabbcb3cd2345fe90ba6e60a651c5a0" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 43445 } } peers { permanent_uuid: "307d9d3a9bfd400c9a4b663f53ac824e" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 39019 } } }
I20260929 18:37:03.341830 19561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 307d9d3a9bfd400c9a4b663f53ac824e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:37:03.341869 19528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b1ad8d91f7945f58403f8f6898f8c7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b1ad8d91f7945f58403f8f6898f8c7f. Latest consensus state: current_term: 1 leader_uuid: "3b1ad8d91f7945f58403f8f6898f8c7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1ad8d91f7945f58403f8f6898f8c7f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43313 } } peers { permanent_uuid: "baabbcb3cd2345fe90ba6e60a651c5a0" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 43445 } } peers { permanent_uuid: "307d9d3a9bfd400c9a4b663f53ac824e" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 39019 } } }
I20260929 18:37:03.343534 19528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b1ad8d91f7945f58403f8f6898f8c7f [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:03.343307 19494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3b1ad8d91f7945f58403f8f6898f8c7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "307d9d3a9bfd400c9a4b663f53ac824e"
I20260929 18:37:03.344930 19494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 307d9d3a9bfd400c9a4b663f53ac824e [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:03.346737 19494 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 307d9d3a9bfd400c9a4b663f53ac824e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b1ad8d91f7945f58403f8f6898f8c7f in term 1.
W20260929 18:37:03.352080 19573 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 307d9d3a9bfd400c9a4b663f53ac824e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:37:03.352208 19573 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:37:03.352576 19574 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:37:03.355890 19574 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:37:03.356972 19374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:03.344315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49712 (local address 127.28.207.126:43313)
0929 18:37:03.344700 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:03.344706 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:03.345296 (+ 590us) server_negotiation.cc:407] Connection header received
0929 18:37:03.345518 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:03.345521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:03.345576 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:03.345627 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:03.346738 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.347526 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.348878 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.349217 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.349294 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:03.351346 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:03.351350 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:03.351353 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:03.351382 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:03.354232 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:03.354814 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:03.354951 (+ 137us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:03.354953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:03.355062 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:03.355533 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:03.355537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:03.355539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:03.355817 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:03.355931 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:03.356604 (+ 673us) server_negotiation.cc:299] Negotiation successful
0929 18:37:03.356798 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280}
I20260929 18:37:03.370934 19450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:03.343370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54904 (local address 127.28.207.125:43445)
0929 18:37:03.343550 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:03.343556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:03.356711 (+ 13155us) server_negotiation.cc:407] Connection header received
0929 18:37:03.356923 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:03.356926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:03.356990 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:03.357045 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:03.361120 (+ 4075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.362152 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.364070 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.364471 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.364576 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:03.366925 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:03.366930 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:03.366933 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:03.366974 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:03.368793 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:03.369383 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:03.369390 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:03.369393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:03.369487 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:03.369806 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:03.369809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:03.369811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:03.370064 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:03.370216 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:03.370576 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 18:37:03.370710 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 18:37:03.378268 19526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:03.345404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33776 (local address 127.28.207.124:39019)
0929 18:37:03.345628 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:03.345632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:03.356501 (+ 10869us) server_negotiation.cc:407] Connection header received
0929 18:37:03.358349 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:03.358353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:03.358422 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:03.358529 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:03.361495 (+ 2966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.362400 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.371856 (+ 9456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:03.372279 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:03.372376 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:03.374330 (+ 1954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:03.374335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:03.374338 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:03.374367 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:03.376113 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:03.376777 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:03.376780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:03.376782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:03.376871 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:03.377163 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:03.377166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:03.377168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:03.377434 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:03.377560 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:03.377930 (+ 370us) server_negotiation.cc:299] Negotiation successful
0929 18:37:03.378069 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260929 18:37:03.384941 19419 raft_consensus.cc:1261] T 00000000000000000000000000000000 P baabbcb3cd2345fe90ba6e60a651c5a0 [term 1 FOLLOWER]: Refusing update from remote peer 3b1ad8d91f7945f58403f8f6898f8c7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:37:03.385092 19494 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 307d9d3a9bfd400c9a4b663f53ac824e [term 1 FOLLOWER]: Refusing update from remote peer 3b1ad8d91f7945f58403f8f6898f8c7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:37:03.385756 19528 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3b1ad8d91f7945f58403f8f6898f8c7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "baabbcb3cd2345fe90ba6e60a651c5a0" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 43445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:37:03.386053 19555 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3b1ad8d91f7945f58403f8f6898f8c7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "307d9d3a9bfd400c9a4b663f53ac824e" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 39019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:37:03.392068 19542 sys_catalog.cc:455] T 00000000000000000000000000000000 P baabbcb3cd2345fe90ba6e60a651c5a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b1ad8d91f7945f58403f8f6898f8c7f. Latest consensus state: current_term: 1 leader_uuid: "3b1ad8d91f7945f58403f8f6898f8c7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1ad8d91f7945f58403f8f6898f8c7f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43313 } } peers { permanent_uuid: "baabbcb3cd2345fe90ba6e60a651c5a0" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 43445 } } peers { permanent_uuid: "307d9d3a9bfd400c9a4b663f53ac824e" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 39019 } } }
I20260929 18:37:03.392241 19542 sys_catalog.cc:458] T 00000000000000000000000000000000 P baabbcb3cd2345fe90ba6e60a651c5a0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:37:03.393182 19561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 307d9d3a9bfd400c9a4b663f53ac824e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b1ad8d91f7945f58403f8f6898f8c7f. Latest consensus state: current_term: 1 leader_uuid: "3b1ad8d91f7945f58403f8f6898f8c7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1ad8d91f7945f58403f8f6898f8c7f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43313 } } peers { permanent_uuid: "baabbcb3cd2345fe90ba6e60a651c5a0" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 43445 } } peers { permanent_uuid: "307d9d3a9bfd400c9a4b663f53ac824e" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 39019 } } }
I20260929 18:37:03.393338 19561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 307d9d3a9bfd400c9a4b663f53ac824e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:37:03.403384 19528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b1ad8d91f7945f58403f8f6898f8c7f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3b1ad8d91f7945f58403f8f6898f8c7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1ad8d91f7945f58403f8f6898f8c7f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43313 } } peers { permanent_uuid: "baabbcb3cd2345fe90ba6e60a651c5a0" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 43445 } } peers { permanent_uuid: "307d9d3a9bfd400c9a4b663f53ac824e" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 39019 } } }
I20260929 18:37:03.403585 19528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b1ad8d91f7945f58403f8f6898f8c7f [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:03.403680 19528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b1ad8d91f7945f58403f8f6898f8c7f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3b1ad8d91f7945f58403f8f6898f8c7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1ad8d91f7945f58403f8f6898f8c7f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43313 } } peers { permanent_uuid: "baabbcb3cd2345fe90ba6e60a651c5a0" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 43445 } } peers { permanent_uuid: "307d9d3a9bfd400c9a4b663f53ac824e" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 39019 } } }
I20260929 18:37:03.403738 19528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b1ad8d91f7945f58403f8f6898f8c7f [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:03.404129 19542 sys_catalog.cc:455] T 00000000000000000000000000000000 P baabbcb3cd2345fe90ba6e60a651c5a0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3b1ad8d91f7945f58403f8f6898f8c7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1ad8d91f7945f58403f8f6898f8c7f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43313 } } peers { permanent_uuid: "baabbcb3cd2345fe90ba6e60a651c5a0" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 43445 } } peers { permanent_uuid: "307d9d3a9bfd400c9a4b663f53ac824e" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 39019 } } }
I20260929 18:37:03.404299 19542 sys_catalog.cc:458] T 00000000000000000000000000000000 P baabbcb3cd2345fe90ba6e60a651c5a0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:37:03.404258 19579 mvcc.cc:205] Tried to move back new op lower bound from 7334735967720321024 to 7334735967594975232. Current Snapshot: MvccSnapshot[applied={T|T < 7334735967720321024}]
I20260929 18:37:03.404702 19580 mvcc.cc:205] Tried to move back new op lower bound from 7334735967720321024 to 7334735967594975232. Current Snapshot: MvccSnapshot[applied={T|T < 7334735967720321024}]
I20260929 18:37:03.405944 19561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 307d9d3a9bfd400c9a4b663f53ac824e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3b1ad8d91f7945f58403f8f6898f8c7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b1ad8d91f7945f58403f8f6898f8c7f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 43313 } } peers { permanent_uuid: "baabbcb3cd2345fe90ba6e60a651c5a0" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 43445 } } peers { permanent_uuid: "307d9d3a9bfd400c9a4b663f53ac824e" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 39019 } } }
I20260929 18:37:03.406347 19561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 307d9d3a9bfd400c9a4b663f53ac824e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:37:03.407374 19581 mvcc.cc:205] Tried to move back new op lower bound from 7334735967720321024 to 7334735967594975232. Current Snapshot: MvccSnapshot[applied={T|T < 7334735967720321024}]
I20260929 18:37:03.415037 19574 catalog_manager.cc:1347] Generated new cluster ID: ad652b9a68ea4679bfb4e3739ec4d6ea
I20260929 18:37:03.415146 19574 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:37:03.442562 19574 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:37:03.444367 19574 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:37:03.461174 19574 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b1ad8d91f7945f58403f8f6898f8c7f: Generated new TSK 0
I20260929 18:37:03.462617 19574 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 18:37:03.502816 19374 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:37:03.499459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49722 (local address 127.28.207.126:43313)
0929 18:37:03.499693 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:03.499699 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:03.501463 (+ 1764us) server_negotiation.cc:407] Connection header received
0929 18:37:03.501722 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:03.501726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:03.501792 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:03.501853 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:03.502639 (+ 786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49722: BlockingRecv error: recv got EOF from 127.0.0.1:49722 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260929 18:37:03.505465 19526 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:37:03.500353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33790 (local address 127.28.207.124:39019)
0929 18:37:03.500489 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:03.500493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:03.504079 (+ 3586us) server_negotiation.cc:407] Connection header received
0929 18:37:03.504308 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:03.504312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:03.504381 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:03.504444 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:03.505313 (+ 869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33790: BlockingRecv error: recv got EOF from 127.0.0.1:33790 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260929 18:37:03.507324 19450 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:37:03.500745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54916 (local address 127.28.207.125:43445)
0929 18:37:03.500889 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:03.500894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:03.506234 (+ 5340us) server_negotiation.cc:407] Connection header received
0929 18:37:03.506433 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:03.506437 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:03.506499 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:03.506558 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:03.507181 (+ 623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54916: BlockingRecv error: recv got EOF from 127.0.0.1:54916 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260929 18:37:03.514154 19450 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:37:03.511232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54920 (local address 127.28.207.125:43445)
0929 18:37:03.511366 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:03.511370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:03.512351 (+ 981us) server_negotiation.cc:407] Connection header received
0929 18:37:03.512630 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:03.512633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:03.512730 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:03.513004 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:03.513839 (+ 835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54920: BlockingRecv error: recv got EOF from 127.0.0.1:54920 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 18:37:03.522084 19450 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:37:03.519926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54928 (local address 127.28.207.125:43445)
0929 18:37:03.520085 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:03.520090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:03.520776 (+ 686us) server_negotiation.cc:407] Connection header received
0929 18:37:03.521030 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:03.521043 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:03.521116 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:03.521177 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:03.521876 (+ 699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54928: BlockingRecv error: recv got EOF from 127.0.0.1:54928 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 18:37:03.530021 19450 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:37:03.527989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54932 (local address 127.28.207.125:43445)
0929 18:37:03.528148 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:03.528153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:03.528768 (+ 615us) server_negotiation.cc:407] Connection header received
0929 18:37:03.529039 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:03.529043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:03.529116 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:03.529194 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:03.529859 (+ 665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54932: BlockingRecv error: recv got EOF from 127.0.0.1:54932 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260929 18:37:03.540030 19450 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:37:03.537720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54944 (local address 127.28.207.125:43445)
0929 18:37:03.537927 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:03.537932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:03.538644 (+ 712us) server_negotiation.cc:407] Connection header received
0929 18:37:03.539018 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:03.539023 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:03.539093 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:03.539161 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:03.539879 (+ 718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54944: BlockingRecv error: recv got EOF from 127.0.0.1:54944 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
W20260929 18:37:03.554682 19450 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:37:03.552422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54950 (local address 127.28.207.125:43445)
0929 18:37:03.552625 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:03.552631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:03.553348 (+ 717us) server_negotiation.cc:407] Connection header received
0929 18:37:03.553625 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:03.553630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:03.553702 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:03.553769 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:03.554517 (+ 748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54950: BlockingRecv error: recv got EOF from 127.0.0.1:54950 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
W20260929 18:37:03.578080 19450 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:37:03.575687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54960 (local address 127.28.207.125:43445)
0929 18:37:03.575869 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:03.575877 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:03.576649 (+ 772us) server_negotiation.cc:407] Connection header received
0929 18:37:03.576907 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:03.576911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:03.576983 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:03.577046 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:03.577894 (+ 848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54960: BlockingRecv error: recv got EOF from 127.0.0.1:54960 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260929 18:37:03.615307 19450 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:37:03.613531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54968 (local address 127.28.207.125:43445)
0929 18:37:03.613746 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:03.613754 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:03.613952 (+ 198us) server_negotiation.cc:407] Connection header received
0929 18:37:03.614163 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:03.614167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:03.614240 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:03.614351 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:03.615118 (+ 767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54968: BlockingRecv error: recv got EOF from 127.0.0.1:54968 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
W20260929 18:37:03.683274 19450 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:37:03.681775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54970 (local address 127.28.207.125:43445)
0929 18:37:03.682018 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:03.682024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:03.682222 (+ 198us) server_negotiation.cc:407] Connection header received
0929 18:37:03.682423 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:03.682431 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:03.682489 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:03.682582 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:03.683121 (+ 539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54970: BlockingRecv error: recv got EOF from 127.0.0.1:54970 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
W20260929 18:37:03.815284 19450 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:37:03.813767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54972 (local address 127.28.207.125:43445)
0929 18:37:03.814034 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:03.814040 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:03.814242 (+ 202us) server_negotiation.cc:407] Connection header received
0929 18:37:03.814438 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:03.814446 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:03.814495 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:03.814592 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:03.815140 (+ 548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54972: BlockingRecv error: recv got EOF from 127.0.0.1:54972 (error 108)
Metrics: {"server-negotiator.queue_time_us":144}
W20260929 18:37:04.075572 19450 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:37:04.074174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54976 (local address 127.28.207.125:43445)
0929 18:37:04.074348 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:04.074352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:04.074445 (+ 93us) server_negotiation.cc:407] Connection header received
0929 18:37:04.074536 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:04.074539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:04.074602 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:04.074713 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:04.075369 (+ 656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54976: BlockingRecv error: recv got EOF from 127.0.0.1:54976 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 18:37:04.343084 19559 catalog_manager.cc:1259] Loaded cluster ID: ad652b9a68ea4679bfb4e3739ec4d6ea
I20260929 18:37:04.343207 19559 catalog_manager.cc:1552] T 00000000000000000000000000000000 P baabbcb3cd2345fe90ba6e60a651c5a0: loading cluster ID for follower catalog manager: success
I20260929 18:37:04.344976 19559 catalog_manager.cc:1574] T 00000000000000000000000000000000 P baabbcb3cd2345fe90ba6e60a651c5a0: acquiring CA information for follower catalog manager: success
I20260929 18:37:04.346220 19559 catalog_manager.cc:1602] T 00000000000000000000000000000000 P baabbcb3cd2345fe90ba6e60a651c5a0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:37:04.353592 19573 catalog_manager.cc:1259] Loaded cluster ID: ad652b9a68ea4679bfb4e3739ec4d6ea
I20260929 18:37:04.353705 19573 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 307d9d3a9bfd400c9a4b663f53ac824e: loading cluster ID for follower catalog manager: success
I20260929 18:37:04.355430 19573 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 307d9d3a9bfd400c9a4b663f53ac824e: acquiring CA information for follower catalog manager: success
I20260929 18:37:04.356619 19573 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 307d9d3a9bfd400c9a4b663f53ac824e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:37:04.594743 19651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:04.590707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54992 (local address 127.28.207.125:43445)
0929 18:37:04.591094 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:04.591097 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:04.591111 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:37:04.591161 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:04.591163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:04.591292 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:04.591437 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:37:04.591694 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:04.592258 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:04.593152 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:04.593386 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:04.593514 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:04.593844 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:37:04.594291 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:37:04.594427 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:04.594493 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 18:37:04.594555 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":177,"threads_started":1}
I20260929 18:37:04.603034 19657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:04.598942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33806 (local address 127.28.207.124:39019)
0929 18:37:04.599278 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:04.599283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:04.599302 (+ 19us) server_negotiation.cc:407] Connection header received
0929 18:37:04.599465 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:04.599467 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:04.599590 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:04.599727 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:37:04.600003 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:04.600542 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:04.601419 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:04.601653 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:04.601766 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:04.602094 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:37:04.602555 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:37:04.602693 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:04.602780 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 18:37:04.602838 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":81,"threads_started":1}
I20260929 18:37:04.612218 19651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:04.607774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55008 (local address 127.28.207.125:43445)
0929 18:37:04.608107 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:04.608113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:04.608132 (+ 19us) server_negotiation.cc:407] Connection header received
0929 18:37:04.608194 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:04.608197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:04.608266 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:04.608391 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:37:04.608685 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:04.609502 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:04.610842 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:04.611222 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:04.611336 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:04.611679 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:37:04.611765 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:37:04.611849 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:04.611996 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 18:37:04.612058 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 18:37:04.613652 19657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:04.608803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33814 (local address 127.28.207.124:39019)
0929 18:37:04.608914 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:04.608920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:04.609523 (+ 603us) server_negotiation.cc:407] Connection header received
0929 18:37:04.609724 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:04.609727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:04.609808 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:04.609892 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:37:04.610567 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:04.611344 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:04.612577 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:04.612818 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:04.612936 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:04.613224 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:37:04.613291 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:37:04.613371 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:04.613467 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 18:37:04.613520 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 18:37:04.614542 19663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:04.607821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49730 (local address 127.28.207.126:43313)
0929 18:37:04.608219 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:04.608223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:04.608783 (+ 560us) server_negotiation.cc:407] Connection header received
0929 18:37:04.608888 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:04.608892 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:04.609053 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:04.609201 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:37:04.609838 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:04.610447 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:04.611750 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:04.612165 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:04.612279 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:04.613087 (+ 808us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:37:04.613571 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:37:04.613674 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:04.614219 (+ 545us) server_negotiation.cc:299] Negotiation successful
0929 18:37:04.614303 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":104,"threads_started":1}
I20260929 18:37:04.619506 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 19302
I20260929 18:37:04.628414 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 19377
I20260929 18:37:04.636646 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 19451
2026-09-29T18:37:04Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3388 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 18:37:04 dist-test-slave-qcpv krb5kdc[19668](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 18:37:04 dist-test-slave-qcpv krb5kdc[19668](info): set up 2 sockets
Sep 29 18:37:04 dist-test-slave-qcpv krb5kdc[19668](info): commencing operation
krb5kdc: starting...
W20260929 18:37:05.731550 29501 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/krb5kdc/test-user.keytab.
Sep 29 18:37:05 dist-test-slave-qcpv krb5kdc[19668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T18:37:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T18:37:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.28.207.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
Entry for principal kudu/127.28.207.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab.
I20260929 18:37:05.887105 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:37125
--webserver_interface=127.28.207.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:43905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:37125,127.28.207.125:39309,127.28.207.124:43105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 18:37:06.013165 19681 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 18:37:06.013586 19681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:06.013679 19681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:06.013778 19681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:06.018257 19681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:37:06.018400 19681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:06.018448 19681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:06.018496 19681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:37:06.018535 19681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:37:06.025076 19681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:43905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:37125,127.28.207.125:39309,127.28.207.124:43105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.126.keytab
--principal=kudu/127.28.207.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.126:37125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.28.207.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:06.026564 19681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:06.027840 19681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:06.038774 19686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:06.038734 19687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:06.038718 19689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:06.039064 19681 server_base.cc:1034] running on GCE node
I20260929 18:37:06.040066 19681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:06.041685 19681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:06.043272 19681 hybrid_clock.cc:648] HybridClock initialized: now 1790707026043187 us; error 96 us; skew 500 ppm
Sep 29 18:37:06 dist-test-slave-qcpv krb5kdc[19668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707026, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:06.047763 19681 init.cc:377] Logged in from keytab as kudu/127.28.207.126@KRBTEST.COM (short username kudu)
I20260929 18:37:06.049266 19681 webserver.cc:460] Webserver started at http://127.28.207.126:43089/ using document root <none> and password file <none>
I20260929 18:37:06.050040 19681 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:06.050119 19681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:06.050381 19681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:06.053156 19681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-0/data/instance:
uuid: "b356e6c8f0f54a728cee031db342674f"
format_stamp: "Formatted at 2026-09-29 18:37:06 on dist-test-slave-qcpv"
server_key: "f3284f8a36a7f99cf6048144821483de"
server_key_iv: "e4d9abffe8fb35abc4412f3235d69b7c"
server_key_version: "encryptionkey@0"
I20260929 18:37:06.053799 19681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-0/wal/instance:
uuid: "b356e6c8f0f54a728cee031db342674f"
format_stamp: "Formatted at 2026-09-29 18:37:06 on dist-test-slave-qcpv"
server_key: "f3284f8a36a7f99cf6048144821483de"
server_key_iv: "e4d9abffe8fb35abc4412f3235d69b7c"
server_key_version: "encryptionkey@0"
I20260929 18:37:06.059628 19681 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260929 18:37:06.063623 19696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:06.065536 19681 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 18:37:06.065716 19681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-0/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-0/wal
uuid: "b356e6c8f0f54a728cee031db342674f"
format_stamp: "Formatted at 2026-09-29 18:37:06 on dist-test-slave-qcpv"
server_key: "f3284f8a36a7f99cf6048144821483de"
server_key_iv: "e4d9abffe8fb35abc4412f3235d69b7c"
server_key_version: "encryptionkey@0"
I20260929 18:37:06.065927 19681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:06.081763 19681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:06.086673 19681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:06.086953 19681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:06.098035 19681 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.126:37125
I20260929 18:37:06.098412 19748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.126:37125 every 8 connection(s)
I20260929 18:37:06.099440 19681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-0/data/info.pb
I20260929 18:37:06.103158 19749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:06.107231 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 19681
I20260929 18:37:06.107385 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-0/wal/instance
I20260929 18:37:06.107667 29501 external_mini_cluster.cc:1428] Setting key d90265a01c8dd3b6dc2eab6ea83ea9f4
I20260929 18:37:06.107739 19749 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.207.126" port: 37125 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.28.207.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.125@KRBTEST.COM" created.
Sep 29 18:37:06 dist-test-slave-qcpv krb5kdc[19668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707026, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:37:06.133982 19752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.111471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:37125 (local address 127.0.0.1:41926)
0929 18:37:06.112212 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:06.112273 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:06.113362 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:06.114086 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:06.114092 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:06.114583 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:06.114831 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:06.114843 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.116082 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.116085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:06.116817 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:06.116826 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.117506 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.117510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:06.117625 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:06.118437 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:06.118453 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:06.124595 (+ 6142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:06.129024 (+ 4429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:06.129038 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:06.129050 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:06.129399 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:06.130333 (+ 934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:06.130336 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:06.130342 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:06.130496 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:06.131751 (+ 1255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:06.131758 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:06.131968 (+ 210us) client_negotiation.cc:769] Sending connection context
0929 18:37:06.132155 (+ 187us) client_negotiation.cc:240] Negotiation successful
0929 18:37:06.132497 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":230,"threads_started":1}
I20260929 18:37:06.133993 19753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.112854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41926 (local address 127.28.207.126:37125)
0929 18:37:06.113241 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:06.113254 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:06.113302 (+ 48us) server_negotiation.cc:407] Connection header received
0929 18:37:06.113683 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:06.113687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:06.113895 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:06.114089 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:06.115027 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.115941 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.116939 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.117310 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.117510 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:06.125100 (+ 7590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:06.125132 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:06.125150 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:06.125197 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:06.128785 (+ 3588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:06.129778 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:06.129791 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:06.129794 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:06.130156 (+ 362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:06.130648 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:06.130652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:06.130655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:06.131544 (+ 889us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:06.131750 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:06.132374 (+ 624us) server_negotiation.cc:299] Negotiation successful
0929 18:37:06.132632 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":149,"threads_started":1}
I20260929 18:37:06.138455 19749 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.207.125" port: 39309 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:37:06.139554 19752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.139217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.125:39309 (local address 127.0.0.1:48750)
0929 18:37:06.139389 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:06.139476 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.125:39309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260929 18:37:06.139883 19700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.207.125:39309: connect: Connection refused (error 111)
I20260929 18:37:06.141137 19752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.140864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.125:39309 (local address 127.0.0.1:48760)
0929 18:37:06.140979 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:06.141030 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.125:39309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 18:37:06.141829 19749 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.207.125:39309: Network error: Client connection negotiation failed: client connection to 127.28.207.125:39309: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.125.keytab.
Entry for principal kudu/127.28.207.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.125.keytab.
I20260929 18:37:06.147655 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.125:39309
--webserver_interface=127.28.207.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.125.keytab
--principal=kudu/127.28.207.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:43905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:37125,127.28.207.125:39309,127.28.207.124:43105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:37:06.191313 19749 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.207.125" port: 39309 } attempt: 1
I20260929 18:37:06.192791 19752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.192315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.125:39309 (local address 127.0.0.1:48776)
0929 18:37:06.192584 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:06.192675 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.125:39309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260929 18:37:06.193925 19752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.193547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.125:39309 (local address 127.0.0.1:48778)
0929 18:37:06.193797 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:06.193861 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.125:39309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
W20260929 18:37:06.194082 19749 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.207.125:39309: Network error: Client connection negotiation failed: client connection to 127.28.207.125:39309: connect: Connection refused (error 111)
I20260929 18:37:06.262306 19749 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.207.125" port: 39309 } attempt: 2
I20260929 18:37:06.263738 19752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.263312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.125:39309 (local address 127.0.0.1:48790)
0929 18:37:06.263543 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:06.263626 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.125:39309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260929 18:37:06.264986 19752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.264614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.125:39309 (local address 127.0.0.1:48794)
0929 18:37:06.264855 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:06.264916 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.125:39309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
W20260929 18:37:06.265285 19749 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.207.125:39309: Network error: Client connection negotiation failed: client connection to 127.28.207.125:39309: connect: Connection refused (error 111)
W20260929 18:37:06.271808 19756 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 18:37:06.272137 19756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:06.272166 19756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:06.272275 19756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 18:37:06.276826 19756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:37:06.276919 19756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:06.276940 19756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:06.276957 19756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:37:06.276973 19756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:37:06.283061 19756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:43905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:37125,127.28.207.125:39309,127.28.207.124:43105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.125.keytab
--principal=kudu/127.28.207.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.125:39309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.28.207.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:06.284248 19756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:06.285301 19756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:06.295171 19761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:06.295187 19762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:06.295625 19756 server_base.cc:1034] running on GCE node
W20260929 18:37:06.295193 19764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:06.296612 19756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:06.298347 19756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:06.299597 19756 hybrid_clock.cc:648] HybridClock initialized: now 1790707026299545 us; error 72 us; skew 500 ppm
Sep 29 18:37:06 dist-test-slave-qcpv krb5kdc[19668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707026, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:06.303171 19756 init.cc:377] Logged in from keytab as kudu/127.28.207.125@KRBTEST.COM (short username kudu)
I20260929 18:37:06.304527 19756 webserver.cc:460] Webserver started at http://127.28.207.125:40573/ using document root <none> and password file <none>
I20260929 18:37:06.305223 19756 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:06.305296 19756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:06.305516 19756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:06.308315 19756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-1/data/instance:
uuid: "d73e16dfb7604f30b4a7692b27be84d4"
format_stamp: "Formatted at 2026-09-29 18:37:06 on dist-test-slave-qcpv"
server_key: "7c45bd91ae69c69ca012c3ba72af0066"
server_key_iv: "488800a5cb1980c0e9df3599fcb45eba"
server_key_version: "encryptionkey@0"
I20260929 18:37:06.309036 19756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-1/wal/instance:
uuid: "d73e16dfb7604f30b4a7692b27be84d4"
format_stamp: "Formatted at 2026-09-29 18:37:06 on dist-test-slave-qcpv"
server_key: "7c45bd91ae69c69ca012c3ba72af0066"
server_key_iv: "488800a5cb1980c0e9df3599fcb45eba"
server_key_version: "encryptionkey@0"
I20260929 18:37:06.314373 19756 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 18:37:06.318228 19771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:06.320103 19756 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 18:37:06.320428 19756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-1/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-1/wal
uuid: "d73e16dfb7604f30b4a7692b27be84d4"
format_stamp: "Formatted at 2026-09-29 18:37:06 on dist-test-slave-qcpv"
server_key: "7c45bd91ae69c69ca012c3ba72af0066"
server_key_iv: "488800a5cb1980c0e9df3599fcb45eba"
server_key_version: "encryptionkey@0"
I20260929 18:37:06.320633 19756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:06.340404 19756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:06.345299 19756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:06.345597 19756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:06.356138 19756 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.125:39309
I20260929 18:37:06.356542 19749 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.207.125" port: 39309 } attempt: 3
I20260929 18:37:06.356563 19823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.125:39309 every 8 connection(s)
I20260929 18:37:06.357806 19756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-1/data/info.pb
I20260929 18:37:06.362174 19824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 18:37:06 dist-test-slave-qcpv krb5kdc[19668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707026, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for kudu/127.28.207.125@KRBTEST.COM
I20260929 18:37:06.367093 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 19756
I20260929 18:37:06.367247 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-1/wal/instance
I20260929 18:37:06.367605 29501 external_mini_cluster.cc:1428] Setting key 566f97bb8443ecb68a38e99058852a4c
I20260929 18:37:06.368072 19824 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.207.126" port: 37125 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:37:06.373121 19752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.357203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.125:39309 (local address 127.0.0.1:48798)
0929 18:37:06.357377 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:06.357395 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:06.357502 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:06.360493 (+ 2991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:06.360498 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:06.360776 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:06.360976 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:06.360987 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.362579 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.362584 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:06.363726 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:06.363742 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.364598 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.364601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:06.364703 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:06.365280 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:06.365375 (+ 95us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:06.367262 (+ 1887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:06.370221 (+ 2959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:06.370227 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:06.370230 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:06.370514 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:06.371224 (+ 710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:06.371231 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:06.371235 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:06.371422 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:06.372467 (+ 1045us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:06.372470 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:06.372586 (+ 116us) client_negotiation.cc:769] Sending connection context
0929 18:37:06.372709 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 18:37:06.372910 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 18:37:06.374322 19825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.358224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48798 (local address 127.28.207.125:39309)
0929 18:37:06.358764 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:06.358784 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:06.358833 (+ 49us) server_negotiation.cc:407] Connection header received
0929 18:37:06.359651 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:06.359671 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:06.360095 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:06.360425 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:06.361275 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.362376 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.363901 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.364439 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.364573 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:06.367556 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:06.367565 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:06.367584 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:06.367625 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:06.370083 (+ 2458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:06.370920 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:06.370933 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:06.370940 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:06.371086 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:06.371598 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:06.371602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:06.371604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:06.372333 (+ 729us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:06.372505 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:06.372915 (+ 410us) server_negotiation.cc:299] Negotiation successful
0929 18:37:06.373302 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":194,"threads_started":1}
I20260929 18:37:06.377283 19749 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.207.124" port: 43105 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:37:06.378393 19752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.377936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:43105 (local address 127.0.0.1:49654)
0929 18:37:06.378135 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:06.378244 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.124:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
Sep 29 18:37:06 dist-test-slave-qcpv krb5kdc[19668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707026, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.125@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.28.207.124@KRBTEST.COM; defaulting to no policy
I20260929 18:37:06.380056 19752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.379703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:43105 (local address 127.0.0.1:49666)
0929 18:37:06.379899 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:06.379952 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.124:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260929 18:37:06.380331 19749 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.207.124:43105: Network error: Client connection negotiation failed: client connection to 127.28.207.124:43105: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.28.207.124@KRBTEST.COM" created.
I20260929 18:37:06.385129 19828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.370739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:37125 (local address 127.0.0.1:41936)
0929 18:37:06.371448 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:06.371502 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:06.371691 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:06.372064 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:06.372074 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:06.372585 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:06.372830 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:06.372843 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.374962 (+ 2119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.374966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:06.375918 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:06.375929 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.376490 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.376494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:06.376633 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:06.377393 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:06.377406 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:06.379915 (+ 2509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:06.382709 (+ 2794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:06.382720 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:06.382726 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:06.383116 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:06.383526 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:06.383529 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:06.383531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:06.383688 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:06.384347 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:06.384364 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:06.384550 (+ 186us) client_negotiation.cc:769] Sending connection context
0929 18:37:06.384749 (+ 199us) client_negotiation.cc:240] Negotiation successful
0929 18:37:06.384963 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":159,"threads_started":1}
I20260929 18:37:06.385326 19753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.370809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41936 (local address 127.28.207.126:37125)
0929 18:37:06.371049 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:06.371057 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:06.371583 (+ 526us) server_negotiation.cc:407] Connection header received
0929 18:37:06.371832 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:06.371836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:06.371897 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:06.371991 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:06.373002 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.374763 (+ 1761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.376069 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.376363 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.376491 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:06.380100 (+ 3609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:06.380107 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:06.380110 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:06.380141 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:06.382471 (+ 2330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:06.383288 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:06.383295 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:06.383300 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:06.383413 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:06.383850 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:06.383854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:06.383855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:06.384213 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:06.384315 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:06.384915 (+ 600us) server_negotiation.cc:299] Negotiation successful
0929 18:37:06.385090 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 18:37:06.386727 19824 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.207.125" port: 39309 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:37:06 dist-test-slave-qcpv krb5kdc[19668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707026, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.125@KRBTEST.COM for kudu/127.28.207.125@KRBTEST.COM
I20260929 18:37:06.401556 19828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.387462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.125:39309 (local address 127.0.0.1:48814)
0929 18:37:06.387680 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:06.387701 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:06.387803 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:06.388187 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:06.388194 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:06.388404 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:06.388572 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:06.388583 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.389544 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.389550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:06.390542 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:06.390556 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.391272 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.391277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:06.391404 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:06.392004 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:06.392025 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:06.396960 (+ 4935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:06.399420 (+ 2460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:06.399423 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:06.399425 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:06.399673 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:06.400141 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:06.400145 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:06.400147 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:06.400326 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:06.400800 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:06.400803 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:06.400920 (+ 117us) client_negotiation.cc:769] Sending connection context
0929 18:37:06.401026 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 18:37:06.401265 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 18:37:06.401583 19825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.387606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48814 (local address 127.28.207.125:39309)
0929 18:37:06.387736 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:06.387740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:06.387754 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:37:06.387929 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:06.387934 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:06.388023 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:06.388119 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:06.388747 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.389396 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.390731 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.391111 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.391222 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:06.397226 (+ 6004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:06.397230 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:06.397233 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:06.397271 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:06.399244 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:06.399917 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:06.399920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:06.399921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:06.399993 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:06.400457 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:06.400460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:06.400461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:06.400674 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:06.400771 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:06.401092 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 18:37:06.401424 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":157,"server-negotiator.queue_time_us":47}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.28.207.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.124.keytab.
Entry for principal kudu/127.28.207.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.124.keytab.
I20260929 18:37:06.402685 19824 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.207.124" port: 43105 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:37:06.403334 29501 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
/tmp/dist-test-taskxlLuiw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.124:43105
--webserver_interface=127.28.207.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.124.keytab
--principal=kudu/127.28.207.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.28.207.84:43905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.28.207.126:37125,127.28.207.125:39309,127.28.207.124:43105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 18:37:06.403759 19828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.403346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:43105 (local address 127.0.0.1:49674)
0929 18:37:06.403568 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:06.403686 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.124:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260929 18:37:06.404305 19773 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.28.207.124:43105: connect: Connection refused (error 111)
I20260929 18:37:06.405578 19828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.405253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:43105 (local address 127.0.0.1:49676)
0929 18:37:06.405428 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:06.405504 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.124:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260929 18:37:06.405822 19824 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.207.124:43105: Network error: Client connection negotiation failed: client connection to 127.28.207.124:43105: connect: Connection refused (error 111)
I20260929 18:37:06.411566 19749 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.207.124" port: 43105 } attempt: 1
I20260929 18:37:06.412578 19752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.412172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:43105 (local address 127.0.0.1:49680)
0929 18:37:06.412420 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:06.412491 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.124:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260929 18:37:06.413569 19752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.413132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:43105 (local address 127.0.0.1:49690)
0929 18:37:06.413371 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:06.413464 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.124:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260929 18:37:06.413849 19749 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.207.124:43105: Network error: Client connection negotiation failed: client connection to 127.28.207.124:43105: connect: Connection refused (error 111)
I20260929 18:37:06.455173 19824 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.207.124" port: 43105 } attempt: 1
I20260929 18:37:06.456477 19828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.455995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:43105 (local address 127.0.0.1:49706)
0929 18:37:06.456252 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:06.456358 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.124:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260929 18:37:06.457527 19828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.457249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:43105 (local address 127.0.0.1:49708)
0929 18:37:06.457378 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:06.457445 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.124:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 18:37:06.457739 19824 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.207.124:43105: Network error: Client connection negotiation failed: client connection to 127.28.207.124:43105: connect: Connection refused (error 111)
I20260929 18:37:06.489110 19749 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.207.124" port: 43105 } attempt: 2
I20260929 18:37:06.490749 19752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.489960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:43105 (local address 127.0.0.1:49710)
0929 18:37:06.490491 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:06.490613 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.124:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260929 18:37:06.491971 19752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.491630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:43105 (local address 127.0.0.1:49726)
0929 18:37:06.491815 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:06.491883 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.124:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260929 18:37:06.492290 19749 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.207.124:43105: Network error: Client connection negotiation failed: client connection to 127.28.207.124:43105: connect: Connection refused (error 111)
W20260929 18:37:06.525267 19831 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 18:37:06.525594 19831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 18:37:06.525620 19831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 18:37:06.525708 19831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 18:37:06.526048 19824 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.207.124" port: 43105 } attempt: 2
I20260929 18:37:06.527237 19828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.526790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:43105 (local address 127.0.0.1:49730)
0929 18:37:06.527028 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:06.527124 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.124:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260929 18:37:06.528204 19828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.527969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:43105 (local address 127.0.0.1:49740)
0929 18:37:06.528087 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:06.528143 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.124:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 18:37:06.528450 19824 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.207.124:43105: Network error: Client connection negotiation failed: client connection to 127.28.207.124:43105: connect: Connection refused (error 111)
W20260929 18:37:06.530609 19831 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 18:37:06.530679 19831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 18:37:06.530700 19831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 18:37:06.530719 19831 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 18:37:06.530735 19831 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 18:37:06.536504 19831 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.28.207.84:43905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.28.207.126:37125,127.28.207.125:39309,127.28.207.124:43105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/krb5kdc/kudu_127.28.207.124.keytab
--principal=kudu/127.28.207.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.28.207.124:43105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.28.207.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 29 Sep 2026 17:59:15 UTC on ccycloud.quasar-vdnleb.root.comops.site
build id 4680908
I20260929 18:37:06.537742 19831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 18:37:06.539065 19831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 18:37:06.549144 19839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:06.549144 19837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 18:37:06.549226 19836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 18:37:06.549388 19831 server_base.cc:1034] running on GCE node
I20260929 18:37:06.550128 19831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 18:37:06.551761 19831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 18:37:06.552961 19831 hybrid_clock.cc:648] HybridClock initialized: now 1790707026552927 us; error 66 us; skew 500 ppm
Sep 29 18:37:06 dist-test-slave-qcpv krb5kdc[19668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707026, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 18:37:06.556761 19831 init.cc:377] Logged in from keytab as kudu/127.28.207.124@KRBTEST.COM (short username kudu)
I20260929 18:37:06.557945 19831 webserver.cc:460] Webserver started at http://127.28.207.124:45285/ using document root <none> and password file <none>
I20260929 18:37:06.558579 19831 fs_manager.cc:359] Metadata directory not provided
I20260929 18:37:06.558655 19831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 18:37:06.558954 19831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 18:37:06.561172 19831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-2/data/instance:
uuid: "545dbbac5a6c46809a4d8c53933228be"
format_stamp: "Formatted at 2026-09-29 18:37:06 on dist-test-slave-qcpv"
server_key: "087d5d76f064a6718edb765a6b2d742c"
server_key_iv: "ca5eba0c6110e39b3b1a41d8bd7e5625"
server_key_version: "encryptionkey@0"
I20260929 18:37:06.561851 19831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-2/wal/instance:
uuid: "545dbbac5a6c46809a4d8c53933228be"
format_stamp: "Formatted at 2026-09-29 18:37:06 on dist-test-slave-qcpv"
server_key: "087d5d76f064a6718edb765a6b2d742c"
server_key_iv: "ca5eba0c6110e39b3b1a41d8bd7e5625"
server_key_version: "encryptionkey@0"
I20260929 18:37:06.566509 19831 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260929 18:37:06.570123 19846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 18:37:06.572163 19831 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 18:37:06.572329 19831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-2/data,/tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-2/wal
uuid: "545dbbac5a6c46809a4d8c53933228be"
format_stamp: "Formatted at 2026-09-29 18:37:06 on dist-test-slave-qcpv"
server_key: "087d5d76f064a6718edb765a6b2d742c"
server_key_iv: "ca5eba0c6110e39b3b1a41d8bd7e5625"
server_key_version: "encryptionkey@0"
I20260929 18:37:06.572504 19831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 18:37:06.591568 19749 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.207.124" port: 43105 } attempt: 3
I20260929 18:37:06.592852 19752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.592393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:43105 (local address 127.0.0.1:49750)
0929 18:37:06.592638 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:06.592737 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.124:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260929 18:37:06.593897 19752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.593642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:43105 (local address 127.0.0.1:49754)
0929 18:37:06.593765 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:06.593830 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.28.207.124:43105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 18:37:06.594061 19749 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.28.207.124:43105: Network error: Client connection negotiation failed: client connection to 127.28.207.124:43105: connect: Connection refused (error 111)
I20260929 18:37:06.602020 19831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 18:37:06.605876 19831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 18:37:06.606099 19831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 18:37:06.614161 19831 rpc_server.cc:307] RPC server started. Bound to: 127.28.207.124:43105
I20260929 18:37:06.614598 19898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.28.207.124:43105 every 8 connection(s)
I20260929 18:37:06.615334 19831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-2/data/info.pb
I20260929 18:37:06.619366 19899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 18:37:06.619696 19824 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.207.124" port: 43105 } attempt: 3
I20260929 18:37:06.623152 29501 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu as pid 19831
I20260929 18:37:06.623332 29501 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxlLuiw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790706718260899-29501-0/minicluster-data/master-2/wal/instance
I20260929 18:37:06.623703 29501 external_mini_cluster.cc:1428] Setting key 2257775cda4e8c5ba4f15c7041075e06
I20260929 18:37:06.627642 19899 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.207.126" port: 37125 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 18:37:06 dist-test-slave-qcpv krb5kdc[19668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707026, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.125@KRBTEST.COM for kudu/127.28.207.124@KRBTEST.COM
Sep 29 18:37:06 dist-test-slave-qcpv krb5kdc[19668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.207.124@KRBTEST.COM
I20260929 18:37:06.637513 19828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.620465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:43105 (local address 127.0.0.1:49766)
0929 18:37:06.620665 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:06.620687 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:06.620816 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:06.624300 (+ 3484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:06.624305 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:06.624561 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:06.624687 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:06.624695 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.626821 (+ 2126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.626826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:06.627714 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:06.627729 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.628878 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.628883 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:06.628985 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:06.629574 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:06.629586 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:06.631285 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:06.634759 (+ 3474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:06.634762 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:06.634765 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:06.635105 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:06.635618 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:06.635625 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:06.635627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:06.635775 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:06.636798 (+ 1023us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:06.636801 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:06.636941 (+ 140us) client_negotiation.cc:769] Sending connection context
0929 18:37:06.637067 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 18:37:06.637271 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
Sep 29 18:37:06 dist-test-slave-qcpv krb5kdc[19668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707026, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.124@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
I20260929 18:37:06.639303 19903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.624537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49768 (local address 127.28.207.124:43105)
0929 18:37:06.625118 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:06.625129 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:06.625187 (+ 58us) server_negotiation.cc:407] Connection header received
0929 18:37:06.625372 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:06.625375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:06.625427 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:06.625542 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:06.626721 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.627349 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.628606 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.628967 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.629082 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:06.633187 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:06.633192 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:06.633198 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:06.633237 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:06.635292 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:06.635937 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:06.635943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:06.635945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:06.636129 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:06.636523 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:06.636526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:06.636527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:06.636801 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:06.637021 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:06.637864 (+ 843us) server_negotiation.cc:299] Negotiation successful
0929 18:37:06.638147 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":106,"server-negotiator.queue_time_us":485,"spinlock_wait_cycles":91904,"thread_start_us":210,"threads_started":1}
I20260929 18:37:06.639313 19901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.621498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49766 (local address 127.28.207.124:43105)
0929 18:37:06.622126 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:06.622151 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:06.622199 (+ 48us) server_negotiation.cc:407] Connection header received
0929 18:37:06.623287 (+ 1088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:06.623318 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:06.623868 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:06.624388 (+ 520us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:06.624915 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.626621 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.627873 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.628718 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.628913 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:06.631508 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:06.631516 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:06.631536 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:06.631566 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:06.634629 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:06.635259 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:06.635268 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:06.635277 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:06.635488 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:06.635943 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:06.635947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:06.635949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:06.636671 (+ 722us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:06.636840 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:06.637272 (+ 432us) server_negotiation.cc:299] Negotiation successful
0929 18:37:06.638054 (+ 782us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":189,"threads_started":1}
I20260929 18:37:06.642684 19904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.630489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.126:37125 (local address 127.0.0.1:41950)
0929 18:37:06.631011 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:06.631083 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:06.631284 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:06.631698 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:06.631708 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:06.632582 (+ 874us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:06.632712 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:06.632718 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.633471 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.633477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:06.634485 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:06.634498 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.635285 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.635288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:06.635403 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:06.636052 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:06.636069 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:06.638571 (+ 2502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:06.640617 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:06.640623 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:06.640629 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:06.640931 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:06.641355 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:06.641357 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:06.641359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:06.641437 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:06.641989 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:06.641996 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:06.642133 (+ 137us) client_negotiation.cc:769] Sending connection context
0929 18:37:06.642297 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 18:37:06.642504 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"spinlock_wait_cycles":15616,"thread_start_us":132,"threads_started":1}
I20260929 18:37:06.643137 19753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.630415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41950 (local address 127.28.207.126:37125)
0929 18:37:06.630569 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:06.630573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:06.631170 (+ 597us) server_negotiation.cc:407] Connection header received
0929 18:37:06.631416 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:06.631419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:06.631470 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:06.631522 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:06.632847 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.633354 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.634631 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.635016 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.635099 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:06.638736 (+ 3637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:06.638740 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:06.638743 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:06.638772 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:06.640488 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:06.641078 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:06.641084 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:06.641086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:06.641230 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:06.641543 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:06.641547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:06.641549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:06.641835 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:06.641980 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:06.642722 (+ 742us) server_negotiation.cc:299] Negotiation successful
0929 18:37:06.642937 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 18:37:06.644888 19824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d73e16dfb7604f30b4a7692b27be84d4: Bootstrap starting.
I20260929 18:37:06.647945 19824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d73e16dfb7604f30b4a7692b27be84d4: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:06.648732 19899 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.207.125" port: 39309 } has no permanent_uuid. Determining permanent_uuid...
I20260929 18:37:06.648854 19824 log.cc:824] T 00000000000000000000000000000000 P d73e16dfb7604f30b4a7692b27be84d4: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:06.651932 19824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d73e16dfb7604f30b4a7692b27be84d4: No bootstrap required, opened a new log
Sep 29 18:37:06 dist-test-slave-qcpv krb5kdc[19668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.207.126@KRBTEST.COM
Sep 29 18:37:06 dist-test-slave-qcpv krb5kdc[19668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707026, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.124@KRBTEST.COM for kudu/127.28.207.125@KRBTEST.COM
I20260929 18:37:06.656167 19824 raft_consensus.cc:348] T 00000000000000000000000000000000 P d73e16dfb7604f30b4a7692b27be84d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b356e6c8f0f54a728cee031db342674f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 37125 } } peers { permanent_uuid: "d73e16dfb7604f30b4a7692b27be84d4" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 39309 } } peers { permanent_uuid: "545dbbac5a6c46809a4d8c53933228be" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 43105 } }
I20260929 18:37:06.656419 19824 raft_consensus.cc:729] T 00000000000000000000000000000000 P d73e16dfb7604f30b4a7692b27be84d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73e16dfb7604f30b4a7692b27be84d4, State: Initialized, Role: FOLLOWER
I20260929 18:37:06.656751 19824 consensus_queue.cc:260] T 00000000000000000000000000000000 P d73e16dfb7604f30b4a7692b27be84d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b356e6c8f0f54a728cee031db342674f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 37125 } } peers { permanent_uuid: "d73e16dfb7604f30b4a7692b27be84d4" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 39309 } } peers { permanent_uuid: "545dbbac5a6c46809a4d8c53933228be" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 43105 } }
I20260929 18:37:06.657261 19906 sys_catalog.cc:455] T 00000000000000000000000000000000 P d73e16dfb7604f30b4a7692b27be84d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b356e6c8f0f54a728cee031db342674f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 37125 } } peers { permanent_uuid: "d73e16dfb7604f30b4a7692b27be84d4" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 39309 } } peers { permanent_uuid: "545dbbac5a6c46809a4d8c53933228be" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 43105 } } }
I20260929 18:37:06.657624 19906 sys_catalog.cc:458] T 00000000000000000000000000000000 P d73e16dfb7604f30b4a7692b27be84d4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:37:06.658300 19824 sys_catalog.cc:564] T 00000000000000000000000000000000 P d73e16dfb7604f30b4a7692b27be84d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:37:06.660836 19904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.649471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.125:39309 (local address 127.0.0.1:48830)
0929 18:37:06.649602 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:06.649620 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:06.649708 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:06.650168 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:06.650171 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:06.650384 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:06.650536 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:06.650544 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.651559 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.651562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:06.652341 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:06.652351 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.653189 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.653194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:06.653336 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:06.653843 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:06.653855 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:06.656448 (+ 2593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:06.658978 (+ 2530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:06.658981 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:06.658984 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:06.659285 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:06.659572 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:06.659577 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:06.659579 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:06.659671 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:06.660166 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:06.660170 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:06.660288 (+ 118us) client_negotiation.cc:769] Sending connection context
0929 18:37:06.660397 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 18:37:06.660594 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 18:37:06.660974 19753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.647256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41952 (local address 127.28.207.126:37125)
0929 18:37:06.647441 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:06.647447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:06.647465 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:37:06.647518 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:06.647521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:06.647577 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:06.647674 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:06.649301 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.650444 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.651529 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.651807 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.651951 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:06.655726 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:06.655734 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:06.655739 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:06.655778 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:06.658014 (+ 2236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:06.658701 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:06.658705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:06.658708 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:06.658832 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:06.659260 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:06.659268 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:06.659271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:06.659619 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:06.660556 (+ 937us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:06.660656 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 18:37:06.660808 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 18:37:06.662093 19825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.649764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48830 (local address 127.28.207.125:39309)
0929 18:37:06.649868 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:06.649873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:06.649891 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:37:06.649924 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:06.649926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:06.649962 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:06.650055 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:06.650673 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.651422 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.652504 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.652827 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.653163 (+ 336us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:06.656670 (+ 3507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:06.656676 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:06.656680 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:06.656717 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:06.658772 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:06.659379 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:06.659385 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:06.659386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:06.659479 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:06.659776 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:06.659780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:06.659781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:06.660036 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:06.660169 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:06.661643 (+ 1474us) server_negotiation.cc:299] Negotiation successful
0929 18:37:06.661940 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 18:37:06.663789 19899 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.28.207.124" port: 43105 } has no permanent_uuid. Determining permanent_uuid...
W20260929 18:37:06.666821 19917 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d73e16dfb7604f30b4a7692b27be84d4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260929 18:37:06.663762 19756 master_runner.cc:186] Error getting master registration for 127.28.207.126:37125: OK, instance_id { permanent_uuid: "b356e6c8f0f54a728cee031db342674f" instance_seqno: 1790707026095020 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 18:37:06.667434 19756 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 18:37:06.667565 19917 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 29 18:37:06 dist-test-slave-qcpv krb5kdc[19668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.28.207.125@KRBTEST.COM
Sep 29 18:37:06 dist-test-slave-qcpv krb5kdc[19668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707026, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.124@KRBTEST.COM for kudu/127.28.207.124@KRBTEST.COM
I20260929 18:37:06.674574 19825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.663086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48838 (local address 127.28.207.125:39309)
0929 18:37:06.663232 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:06.663236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:06.663256 (+ 20us) server_negotiation.cc:407] Connection header received
0929 18:37:06.663302 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:06.663305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:06.663360 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:06.663451 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:06.664403 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.665257 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.666522 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.666911 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.669357 (+ 2446us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:06.670586 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:06.670590 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:06.670596 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:06.670632 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:06.672751 (+ 2119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:06.673219 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:06.673223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:06.673225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:06.673314 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:06.673639 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:06.673643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:06.673648 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:06.673854 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:06.674127 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:06.674238 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 18:37:06.674366 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 18:37:06.677667 19904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.664330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:43105 (local address 127.0.0.1:49770)
0929 18:37:06.664611 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:06.664625 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:06.664705 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:06.667428 (+ 2723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:06.667431 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:06.667661 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:06.667753 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:06.667760 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.669119 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.669125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:06.670023 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:06.670036 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.670672 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.670679 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:06.670802 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:06.671847 (+ 1045us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:06.671859 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:06.673752 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:06.675882 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:06.675885 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:06.675888 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:06.676121 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:06.676421 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:06.676424 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:06.676426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:06.676584 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:06.677080 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:06.677083 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:06.677175 (+ 92us) client_negotiation.cc:769] Sending connection context
0929 18:37:06.677288 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 18:37:06.677477 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215}
I20260929 18:37:06.678525 19903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.664510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49770 (local address 127.28.207.124:43105)
0929 18:37:06.667110 (+ 2600us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:06.667115 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:06.667131 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:37:06.667169 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:06.667171 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:06.667220 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:06.667307 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:06.667938 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.668955 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.670167 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.670541 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.670658 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:06.673904 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:06.673908 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:06.673910 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:06.673934 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:06.675765 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:06.676224 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:06.676227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:06.676229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:06.676301 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:06.676696 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:06.676700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:06.676701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:06.676973 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:06.677089 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:06.677400 (+ 311us) server_negotiation.cc:299] Negotiation successful
0929 18:37:06.678312 (+ 912us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2517}
I20260929 18:37:06.679210 29501 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 18:37:06.680848 19723 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b356e6c8f0f54a728cee031db342674f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41952
I20260929 18:37:06.682212 19899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 545dbbac5a6c46809a4d8c53933228be: Bootstrap starting.
I20260929 18:37:06.685556 19899 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 545dbbac5a6c46809a4d8c53933228be: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:06.686564 19899 log.cc:824] T 00000000000000000000000000000000 P 545dbbac5a6c46809a4d8c53933228be: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:06.689162 19899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 545dbbac5a6c46809a4d8c53933228be: No bootstrap required, opened a new log
I20260929 18:37:06.693413 19899 raft_consensus.cc:348] T 00000000000000000000000000000000 P 545dbbac5a6c46809a4d8c53933228be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b356e6c8f0f54a728cee031db342674f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 37125 } } peers { permanent_uuid: "d73e16dfb7604f30b4a7692b27be84d4" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 39309 } } peers { permanent_uuid: "545dbbac5a6c46809a4d8c53933228be" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 43105 } }
I20260929 18:37:06.693737 19899 raft_consensus.cc:729] T 00000000000000000000000000000000 P 545dbbac5a6c46809a4d8c53933228be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 545dbbac5a6c46809a4d8c53933228be, State: Initialized, Role: FOLLOWER
I20260929 18:37:06.694272 19899 consensus_queue.cc:260] T 00000000000000000000000000000000 P 545dbbac5a6c46809a4d8c53933228be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b356e6c8f0f54a728cee031db342674f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 37125 } } peers { permanent_uuid: "d73e16dfb7604f30b4a7692b27be84d4" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 39309 } } peers { permanent_uuid: "545dbbac5a6c46809a4d8c53933228be" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 43105 } }
I20260929 18:37:06.695333 19920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 545dbbac5a6c46809a4d8c53933228be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b356e6c8f0f54a728cee031db342674f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 37125 } } peers { permanent_uuid: "d73e16dfb7604f30b4a7692b27be84d4" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 39309 } } peers { permanent_uuid: "545dbbac5a6c46809a4d8c53933228be" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 43105 } } }
I20260929 18:37:06.695487 19920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 545dbbac5a6c46809a4d8c53933228be [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:37:06.696401 19899 sys_catalog.cc:564] T 00000000000000000000000000000000 P 545dbbac5a6c46809a4d8c53933228be [sys.catalog]: configured and running, proceeding with master startup.
W20260929 18:37:06.703123 19931 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 545dbbac5a6c46809a4d8c53933228be: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:37:06.711118 19931 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:37:06.712469 19831 master_runner.cc:186] Error getting master registration for 127.28.207.126:37125: OK, instance_id { permanent_uuid: "b356e6c8f0f54a728cee031db342674f" instance_seqno: 1790707026095020 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 18:37:06.716770 19831 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:37:06.758221 19749 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.28.207.124" port: 43105 } attempt: 4
Sep 29 18:37:06 dist-test-slave-qcpv krb5kdc[19668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790707026, etypes {rep=17 tkt=17 ses=17}, kudu/127.28.207.126@KRBTEST.COM for kudu/127.28.207.124@KRBTEST.COM
I20260929 18:37:06.771198 19752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.759159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.28.207.124:43105 (local address 127.0.0.1:49772)
0929 18:37:06.759420 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 18:37:06.759437 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 18:37:06.759566 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 18:37:06.760220 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 18:37:06.760224 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 18:37:06.760497 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0929 18:37:06.760626 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:06.760634 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.761774 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.761777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:06.763048 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 18:37:06.763063 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.763929 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.763933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 18:37:06.764030 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:06.764822 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 18:37:06.764839 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 18:37:06.766758 (+ 1919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 18:37:06.769082 (+ 2324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:06.769086 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:06.769190 (+ 104us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:06.769476 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:06.769814 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 18:37:06.769818 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 18:37:06.769820 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 18:37:06.769908 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 18:37:06.770542 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 18:37:06.770545 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 18:37:06.770658 (+ 113us) client_negotiation.cc:769] Sending connection context
0929 18:37:06.770758 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 18:37:06.770960 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260929 18:37:06.771756 19903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.759254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49772 (local address 127.28.207.124:43105)
0929 18:37:06.759407 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:06.759412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:06.759545 (+ 133us) server_negotiation.cc:407] Connection header received
0929 18:37:06.759858 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:06.759863 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:06.759956 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:06.760060 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:06.760788 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.761636 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.763220 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.763636 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.763750 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:06.766994 (+ 3244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:06.766999 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:06.767003 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:06.767036 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:06.768933 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:06.769611 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:06.769615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:06.769618 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:06.769713 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:06.770003 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:06.770006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:06.770008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:06.770407 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:06.770501 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:06.771453 (+ 952us) server_negotiation.cc:299] Negotiation successful
0929 18:37:06.771589 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 18:37:06.776425 19749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b356e6c8f0f54a728cee031db342674f: Bootstrap starting.
I20260929 18:37:06.780237 19749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b356e6c8f0f54a728cee031db342674f: Neither blocks nor log segments found. Creating new log.
I20260929 18:37:06.781543 19749 log.cc:824] T 00000000000000000000000000000000 P b356e6c8f0f54a728cee031db342674f: Log is configured to *not* fsync() on all Append() calls
I20260929 18:37:06.784497 19749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b356e6c8f0f54a728cee031db342674f: No bootstrap required, opened a new log
I20260929 18:37:06.788769 19749 raft_consensus.cc:348] T 00000000000000000000000000000000 P b356e6c8f0f54a728cee031db342674f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b356e6c8f0f54a728cee031db342674f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 37125 } } peers { permanent_uuid: "d73e16dfb7604f30b4a7692b27be84d4" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 39309 } } peers { permanent_uuid: "545dbbac5a6c46809a4d8c53933228be" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 43105 } }
I20260929 18:37:06.789089 19749 raft_consensus.cc:729] T 00000000000000000000000000000000 P b356e6c8f0f54a728cee031db342674f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b356e6c8f0f54a728cee031db342674f, State: Initialized, Role: FOLLOWER
I20260929 18:37:06.789485 19749 consensus_queue.cc:260] T 00000000000000000000000000000000 P b356e6c8f0f54a728cee031db342674f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b356e6c8f0f54a728cee031db342674f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 37125 } } peers { permanent_uuid: "d73e16dfb7604f30b4a7692b27be84d4" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 39309 } } peers { permanent_uuid: "545dbbac5a6c46809a4d8c53933228be" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 43105 } }
I20260929 18:37:06.790292 19934 sys_catalog.cc:455] T 00000000000000000000000000000000 P b356e6c8f0f54a728cee031db342674f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b356e6c8f0f54a728cee031db342674f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 37125 } } peers { permanent_uuid: "d73e16dfb7604f30b4a7692b27be84d4" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 39309 } } peers { permanent_uuid: "545dbbac5a6c46809a4d8c53933228be" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 43105 } } }
I20260929 18:37:06.790750 19934 sys_catalog.cc:458] T 00000000000000000000000000000000 P b356e6c8f0f54a728cee031db342674f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:37:06.791369 19749 sys_catalog.cc:564] T 00000000000000000000000000000000 P b356e6c8f0f54a728cee031db342674f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 18:37:06.791483 19723 raft_consensus.cc:482] T 00000000000000000000000000000000 P b356e6c8f0f54a728cee031db342674f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 18:37:06.791615 19723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b356e6c8f0f54a728cee031db342674f [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:06.793253 19723 raft_consensus.cc:504] T 00000000000000000000000000000000 P b356e6c8f0f54a728cee031db342674f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b356e6c8f0f54a728cee031db342674f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 37125 } } peers { permanent_uuid: "d73e16dfb7604f30b4a7692b27be84d4" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 39309 } } peers { permanent_uuid: "545dbbac5a6c46809a4d8c53933228be" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 43105 } }
I20260929 18:37:06.795456 19798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b356e6c8f0f54a728cee031db342674f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d73e16dfb7604f30b4a7692b27be84d4"
I20260929 18:37:06.795785 19798 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d73e16dfb7604f30b4a7692b27be84d4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:06.796001 19873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b356e6c8f0f54a728cee031db342674f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "545dbbac5a6c46809a4d8c53933228be"
I20260929 18:37:06.796310 19873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 545dbbac5a6c46809a4d8c53933228be [term 0 FOLLOWER]: Advancing to term 1
I20260929 18:37:06.797571 19798 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d73e16dfb7604f30b4a7692b27be84d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b356e6c8f0f54a728cee031db342674f in term 1.
I20260929 18:37:06.797932 19873 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 545dbbac5a6c46809a4d8c53933228be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b356e6c8f0f54a728cee031db342674f in term 1.
I20260929 18:37:06.798177 19700 leader_election.cc:302] T 00000000000000000000000000000000 P b356e6c8f0f54a728cee031db342674f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b356e6c8f0f54a728cee031db342674f, d73e16dfb7604f30b4a7692b27be84d4; no voters:
W20260929 18:37:06.802577 19945 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b356e6c8f0f54a728cee031db342674f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 18:37:06.802685 19945 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 18:37:06.805286 19681 master_runner.cc:263] No leader master found from master b356e6c8f0f54a728cee031db342674f
I20260929 18:37:06.805398 19681 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 18:37:06.805534 19723 leader_election.cc:288] T 00000000000000000000000000000000 P b356e6c8f0f54a728cee031db342674f [CANDIDATE]: Term 1 election: Requested vote from peers d73e16dfb7604f30b4a7692b27be84d4 (127.28.207.125:39309), 545dbbac5a6c46809a4d8c53933228be (127.28.207.124:43105)
I20260929 18:37:06.806263 19934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b356e6c8f0f54a728cee031db342674f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 18:37:06.806792 19934 raft_consensus.cc:686] T 00000000000000000000000000000000 P b356e6c8f0f54a728cee031db342674f [term 1 LEADER]: Becoming Leader. State: Replica: b356e6c8f0f54a728cee031db342674f, State: Running, Role: LEADER
I20260929 18:37:06.807299 19934 consensus_queue.cc:237] T 00000000000000000000000000000000 P b356e6c8f0f54a728cee031db342674f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b356e6c8f0f54a728cee031db342674f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 37125 } } peers { permanent_uuid: "d73e16dfb7604f30b4a7692b27be84d4" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 39309 } } peers { permanent_uuid: "545dbbac5a6c46809a4d8c53933228be" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 43105 } }
I20260929 18:37:06.809635 19934 sys_catalog.cc:455] T 00000000000000000000000000000000 P b356e6c8f0f54a728cee031db342674f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b356e6c8f0f54a728cee031db342674f. Latest consensus state: current_term: 1 leader_uuid: "b356e6c8f0f54a728cee031db342674f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b356e6c8f0f54a728cee031db342674f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 37125 } } peers { permanent_uuid: "d73e16dfb7604f30b4a7692b27be84d4" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 39309 } } peers { permanent_uuid: "545dbbac5a6c46809a4d8c53933228be" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 43105 } } }
I20260929 18:37:06.809839 19934 sys_catalog.cc:458] T 00000000000000000000000000000000 P b356e6c8f0f54a728cee031db342674f [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:06.810304 19953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 18:37:06.811524 19953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 18:37:06.819437 19798 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d73e16dfb7604f30b4a7692b27be84d4 [term 1 FOLLOWER]: Refusing update from remote peer b356e6c8f0f54a728cee031db342674f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:37:06.820672 19873 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 545dbbac5a6c46809a4d8c53933228be [term 1 FOLLOWER]: Refusing update from remote peer b356e6c8f0f54a728cee031db342674f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 18:37:06.821509 19947 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b356e6c8f0f54a728cee031db342674f [LEADER]: Connected to new peer: Peer: permanent_uuid: "545dbbac5a6c46809a4d8c53933228be" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 43105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:37:06.821780 19934 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b356e6c8f0f54a728cee031db342674f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73e16dfb7604f30b4a7692b27be84d4" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 18:37:06.829092 19906 sys_catalog.cc:455] T 00000000000000000000000000000000 P d73e16dfb7604f30b4a7692b27be84d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b356e6c8f0f54a728cee031db342674f. Latest consensus state: current_term: 1 leader_uuid: "b356e6c8f0f54a728cee031db342674f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b356e6c8f0f54a728cee031db342674f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 37125 } } peers { permanent_uuid: "d73e16dfb7604f30b4a7692b27be84d4" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 39309 } } peers { permanent_uuid: "545dbbac5a6c46809a4d8c53933228be" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 43105 } } }
I20260929 18:37:06.829262 19906 sys_catalog.cc:458] T 00000000000000000000000000000000 P d73e16dfb7604f30b4a7692b27be84d4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:37:06.831904 19920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 545dbbac5a6c46809a4d8c53933228be [sys.catalog]: SysCatalogTable state changed. Reason: New leader b356e6c8f0f54a728cee031db342674f. Latest consensus state: current_term: 1 leader_uuid: "b356e6c8f0f54a728cee031db342674f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b356e6c8f0f54a728cee031db342674f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 37125 } } peers { permanent_uuid: "d73e16dfb7604f30b4a7692b27be84d4" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 39309 } } peers { permanent_uuid: "545dbbac5a6c46809a4d8c53933228be" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 43105 } } }
I20260929 18:37:06.832082 19920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 545dbbac5a6c46809a4d8c53933228be [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:37:06.835340 19825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.815614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48844 (local address 127.28.207.125:39309)
0929 18:37:06.815798 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:06.815803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:06.818305 (+ 2502us) server_negotiation.cc:407] Connection header received
0929 18:37:06.824967 (+ 6662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:06.824971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:06.825039 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:06.825098 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:06.826382 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.827239 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.828495 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.828860 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.828950 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:06.831173 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:06.831178 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:06.831181 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:06.831218 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:06.833086 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:06.833665 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:06.833670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:06.833671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:06.833765 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:06.834103 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:06.834107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:06.834110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:06.834370 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:06.834535 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:06.834968 (+ 433us) server_negotiation.cc:299] Negotiation successful
0929 18:37:06.835132 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 18:37:06.838605 19903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.817529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49786 (local address 127.28.207.124:43105)
0929 18:37:06.817675 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:06.817681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:06.823227 (+ 5546us) server_negotiation.cc:407] Connection header received
0929 18:37:06.823454 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:06.823457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:06.823522 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:06.823581 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:06.824750 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.825732 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.832162 (+ 6430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.832583 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.832679 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:06.834503 (+ 1824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:06.834507 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:06.834509 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:06.834561 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:06.836643 (+ 2082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:06.837163 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:06.837167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:06.837169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:06.837258 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:06.837584 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:06.837587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:06.837589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:06.837820 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:06.837922 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:06.838298 (+ 376us) server_negotiation.cc:299] Negotiation successful
0929 18:37:06.838425 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 18:37:06.838626 19947 sys_catalog.cc:455] T 00000000000000000000000000000000 P b356e6c8f0f54a728cee031db342674f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b356e6c8f0f54a728cee031db342674f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b356e6c8f0f54a728cee031db342674f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 37125 } } peers { permanent_uuid: "d73e16dfb7604f30b4a7692b27be84d4" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 39309 } } peers { permanent_uuid: "545dbbac5a6c46809a4d8c53933228be" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 43105 } } }
I20260929 18:37:06.838794 19947 sys_catalog.cc:458] T 00000000000000000000000000000000 P b356e6c8f0f54a728cee031db342674f [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:06.839401 19920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 545dbbac5a6c46809a4d8c53933228be [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b356e6c8f0f54a728cee031db342674f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b356e6c8f0f54a728cee031db342674f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 37125 } } peers { permanent_uuid: "d73e16dfb7604f30b4a7692b27be84d4" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 39309 } } peers { permanent_uuid: "545dbbac5a6c46809a4d8c53933228be" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 43105 } } }
I20260929 18:37:06.839517 19920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 545dbbac5a6c46809a4d8c53933228be [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:37:06.840262 19954 mvcc.cc:205] Tried to move back new op lower bound from 7334735981842214912 to 7334735981807558656. Current Snapshot: MvccSnapshot[applied={T|T < 7334735981842214912}]
I20260929 18:37:06.843588 19934 sys_catalog.cc:455] T 00000000000000000000000000000000 P b356e6c8f0f54a728cee031db342674f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b356e6c8f0f54a728cee031db342674f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b356e6c8f0f54a728cee031db342674f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 37125 } } peers { permanent_uuid: "d73e16dfb7604f30b4a7692b27be84d4" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 39309 } } peers { permanent_uuid: "545dbbac5a6c46809a4d8c53933228be" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 43105 } } }
I20260929 18:37:06.843748 19934 sys_catalog.cc:458] T 00000000000000000000000000000000 P b356e6c8f0f54a728cee031db342674f [sys.catalog]: This master's current role is: LEADER
I20260929 18:37:06.845180 19906 sys_catalog.cc:455] T 00000000000000000000000000000000 P d73e16dfb7604f30b4a7692b27be84d4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b356e6c8f0f54a728cee031db342674f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b356e6c8f0f54a728cee031db342674f" member_type: VOTER last_known_addr { host: "127.28.207.126" port: 37125 } } peers { permanent_uuid: "d73e16dfb7604f30b4a7692b27be84d4" member_type: VOTER last_known_addr { host: "127.28.207.125" port: 39309 } } peers { permanent_uuid: "545dbbac5a6c46809a4d8c53933228be" member_type: VOTER last_known_addr { host: "127.28.207.124" port: 43105 } } }
I20260929 18:37:06.845322 19906 sys_catalog.cc:458] T 00000000000000000000000000000000 P d73e16dfb7604f30b4a7692b27be84d4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 18:37:06.845616 19753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:06.816746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41962 (local address 127.28.207.126:37125)
0929 18:37:06.816897 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:06.816902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:06.822468 (+ 5566us) server_negotiation.cc:407] Connection header received
0929 18:37:06.822682 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:06.822685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:06.822735 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:06.822784 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:06.825964 (+ 3180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.826718 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.839368 (+ 12650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:06.839698 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:06.839780 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:06.841848 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 18:37:06.841853 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 18:37:06.841856 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 18:37:06.841886 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 18:37:06.843637 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:06.844167 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:06.844171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:06.844173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:06.844265 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 18:37:06.844574 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 18:37:06.844577 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 18:37:06.844579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 18:37:06.844811 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 18:37:06.844934 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:06.845315 (+ 381us) server_negotiation.cc:299] Negotiation successful
0929 18:37:06.845445 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 18:37:06.846410 19960 mvcc.cc:205] Tried to move back new op lower bound from 7334735981842214912 to 7334735981807558656. Current Snapshot: MvccSnapshot[applied={T|T < 7334735981842214912}]
I20260929 18:37:06.846660 19958 mvcc.cc:205] Tried to move back new op lower bound from 7334735981842214912 to 7334735981807558656. Current Snapshot: MvccSnapshot[applied={T|T < 7334735981842214912}]
I20260929 18:37:06.847986 19953 catalog_manager.cc:1347] Generated new cluster ID: fd89cfaf10a547eb96dd113042a7c393
I20260929 18:37:06.848057 19953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 18:37:06.870606 19953 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 18:37:06.872160 19953 catalog_manager.cc:1504] Loading token signing keys...
I20260929 18:37:06.893687 19953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b356e6c8f0f54a728cee031db342674f: Generated new TSK 0
I20260929 18:37:06.894532 19953 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 18:37:06.925961 19753 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:37:06.922747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41976 (local address 127.28.207.126:37125)
0929 18:37:06.922923 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:06.922928 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:06.923610 (+ 682us) server_negotiation.cc:407] Connection header received
0929 18:37:06.924342 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:06.924348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:06.924429 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:06.924554 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:06.925776 (+ 1222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41976: BlockingRecv error: recv got EOF from 127.0.0.1:41976 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260929 18:37:06.925995 19903 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:37:06.924029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49794 (local address 127.28.207.124:43105)
0929 18:37:06.924188 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:06.924193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:06.924800 (+ 607us) server_negotiation.cc:407] Connection header received
0929 18:37:06.924996 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:06.925000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:06.925081 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:06.925144 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:06.925836 (+ 692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49794: BlockingRecv error: recv got EOF from 127.0.0.1:49794 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260929 18:37:06.927415 19825 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:37:06.922763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48850 (local address 127.28.207.125:39309)
0929 18:37:06.922986 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:06.922992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:06.924671 (+ 1679us) server_negotiation.cc:407] Connection header received
0929 18:37:06.924731 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:06.924734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:06.924789 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:06.924871 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:06.927270 (+ 2399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48850: BlockingRecv error: recv got EOF from 127.0.0.1:48850 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260929 18:37:06.931397 19825 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:37:06.929235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48864 (local address 127.28.207.125:39309)
0929 18:37:06.929501 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:06.929506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:06.930175 (+ 669us) server_negotiation.cc:407] Connection header received
0929 18:37:06.930474 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:06.930478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:06.930575 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:06.930690 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:06.931269 (+ 579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48864: BlockingRecv error: recv got EOF from 127.0.0.1:48864 (error 108)
Metrics: {"server-negotiator.queue_time_us":167}
W20260929 18:37:06.936494 19825 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:37:06.935423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48868 (local address 127.28.207.125:39309)
0929 18:37:06.935587 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:06.935592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:06.935608 (+ 16us) server_negotiation.cc:407] Connection header received
0929 18:37:06.935674 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:06.935677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:06.935740 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:06.935850 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:06.936361 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48868: BlockingRecv error: recv got EOF from 127.0.0.1:48868 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 18:37:06.945042 19825 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:37:06.943542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48882 (local address 127.28.207.125:39309)
0929 18:37:06.943688 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:06.943693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:06.943976 (+ 283us) server_negotiation.cc:407] Connection header received
0929 18:37:06.944168 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:06.944172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:06.944234 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:06.944354 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:06.944906 (+ 552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48882: BlockingRecv error: recv got EOF from 127.0.0.1:48882 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260929 18:37:06.953616 19825 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:37:06.952096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48894 (local address 127.28.207.125:39309)
0929 18:37:06.952317 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:06.952323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:06.952536 (+ 213us) server_negotiation.cc:407] Connection header received
0929 18:37:06.952707 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:06.952711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:06.952784 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:06.952927 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:06.953461 (+ 534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48894: BlockingRecv error: recv got EOF from 127.0.0.1:48894 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
W20260929 18:37:06.966944 19825 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:37:06.965148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48896 (local address 127.28.207.125:39309)
0929 18:37:06.965289 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:06.965295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:06.965664 (+ 369us) server_negotiation.cc:407] Connection header received
0929 18:37:06.965825 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:06.965830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:06.965921 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:06.966043 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:06.966741 (+ 698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48896: BlockingRecv error: recv got EOF from 127.0.0.1:48896 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 18:37:06.986279 19825 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:37:06.985043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48912 (local address 127.28.207.125:39309)
0929 18:37:06.985262 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:06.985268 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:06.985287 (+ 19us) server_negotiation.cc:407] Connection header received
0929 18:37:06.985449 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:06.985453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:06.985517 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:06.985615 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:06.986115 (+ 500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48912: BlockingRecv error: recv got EOF from 127.0.0.1:48912 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260929 18:37:07.022209 19825 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:37:07.020899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48928 (local address 127.28.207.125:39309)
0929 18:37:07.021040 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:07.021047 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:07.021283 (+ 236us) server_negotiation.cc:407] Connection header received
0929 18:37:07.021468 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:07.021476 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:07.021526 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:07.021620 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:07.022065 (+ 445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48928: BlockingRecv error: recv got EOF from 127.0.0.1:48928 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 18:37:07.090133 19825 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:37:07.088800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48930 (local address 127.28.207.125:39309)
0929 18:37:07.089017 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:07.089023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:07.089196 (+ 173us) server_negotiation.cc:407] Connection header received
0929 18:37:07.089386 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:07.089389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:07.089439 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:07.089531 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:07.089994 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48930: BlockingRecv error: recv got EOF from 127.0.0.1:48930 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
W20260929 18:37:07.223009 19825 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:37:07.221084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48944 (local address 127.28.207.125:39309)
0929 18:37:07.221327 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:07.221333 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:07.221467 (+ 134us) server_negotiation.cc:407] Connection header received
0929 18:37:07.221684 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:07.221688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:07.221754 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:07.222068 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:07.222711 (+ 643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48944: BlockingRecv error: recv got EOF from 127.0.0.1:48944 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
W20260929 18:37:07.483367 19825 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 18:37:07.481765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48948 (local address 127.28.207.125:39309)
0929 18:37:07.481983 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:07.481989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:07.482010 (+ 21us) server_negotiation.cc:407] Connection header received
0929 18:37:07.482256 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:07.482260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:07.482333 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:07.482426 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 18:37:07.483189 (+ 763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48948: BlockingRecv error: recv got EOF from 127.0.0.1:48948 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
I20260929 18:37:07.669135 19917 catalog_manager.cc:1259] Loaded cluster ID: fd89cfaf10a547eb96dd113042a7c393
I20260929 18:37:07.669274 19917 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d73e16dfb7604f30b4a7692b27be84d4: loading cluster ID for follower catalog manager: success
I20260929 18:37:07.671262 19917 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d73e16dfb7604f30b4a7692b27be84d4: acquiring CA information for follower catalog manager: success
I20260929 18:37:07.672449 19917 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d73e16dfb7604f30b4a7692b27be84d4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:37:07.712612 19931 catalog_manager.cc:1259] Loaded cluster ID: fd89cfaf10a547eb96dd113042a7c393
I20260929 18:37:07.712759 19931 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 545dbbac5a6c46809a4d8c53933228be: loading cluster ID for follower catalog manager: success
I20260929 18:37:07.714404 19931 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 545dbbac5a6c46809a4d8c53933228be: acquiring CA information for follower catalog manager: success
I20260929 18:37:07.715593 19931 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 545dbbac5a6c46809a4d8c53933228be: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 18:37:08.002643 20029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:07.997677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48964 (local address 127.28.207.125:39309)
0929 18:37:07.997954 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:07.997958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:07.998248 (+ 290us) server_negotiation.cc:407] Connection header received
0929 18:37:07.998376 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:07.998379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:07.998516 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:07.998642 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:37:07.999098 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:07.999659 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:08.000806 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:08.001052 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:08.001154 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:08.001573 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:37:08.002070 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:37:08.002171 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:08.002341 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 18:37:08.002404 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":76,"threads_started":1}
I20260929 18:37:08.010697 20035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:08.006457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49798 (local address 127.28.207.124:43105)
0929 18:37:08.006809 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:08.006814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:08.006828 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:37:08.006984 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:08.006987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:08.007111 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:08.007240 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:37:08.007487 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:08.008046 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:08.009126 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:08.009368 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:08.009482 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:08.009818 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:37:08.010282 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:37:08.010420 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:08.010463 (+ 43us) server_negotiation.cc:299] Negotiation successful
0929 18:37:08.010523 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":92,"threads_started":1}
I20260929 18:37:08.019747 20035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:08.015259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49812 (local address 127.28.207.124:43105)
0929 18:37:08.015418 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:08.015423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:08.015441 (+ 18us) server_negotiation.cc:407] Connection header received
0929 18:37:08.015527 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:08.015531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:08.015607 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:08.015690 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:37:08.016152 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:08.017162 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:08.018403 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:08.018655 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:08.018786 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:08.019152 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:37:08.019249 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:37:08.019337 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:08.019492 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 18:37:08.019587 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 18:37:08.019822 20029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:08.014739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48968 (local address 127.28.207.125:39309)
0929 18:37:08.014943 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:08.014949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:08.015630 (+ 681us) server_negotiation.cc:407] Connection header received
0929 18:37:08.015735 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:08.015738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:08.015816 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:08.015914 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:37:08.016245 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:08.016890 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:08.018258 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:08.018626 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:08.018737 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:08.019099 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:37:08.019201 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:37:08.019355 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:08.019533 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 18:37:08.019625 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 18:37:08.020686 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 18:37:08.014445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41984 (local address 127.28.207.126:37125)
0929 18:37:08.015645 (+ 1200us) server_negotiation.cc:206] Beginning negotiation
0929 18:37:08.015651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 18:37:08.015665 (+ 14us) server_negotiation.cc:407] Connection header received
0929 18:37:08.015773 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 18:37:08.015776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 18:37:08.016016 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 18:37:08.016168 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 18:37:08.016434 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:08.017402 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:08.018590 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 18:37:08.019019 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 18:37:08.019120 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 18:37:08.019725 (+ 605us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 18:37:08.020108 (+ 383us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 18:37:08.020193 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 18:37:08.020402 (+ 209us) server_negotiation.cc:299] Negotiation successful
0929 18:37:08.020499 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1097,"thread_start_us":282,"threads_started":1}
I20260929 18:37:08.031198 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 19681
I20260929 18:37:08.039506 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 19756
I20260929 18:37:08.047062 29501 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxlLuiw/build/debug/bin/kudu with pid 19831
2026-09-29T18:37:08Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3409 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6797 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (309778 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST